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ABSTRACT 


This  thesis  develops  a  quantitative  process  for  determining  if  the  changes  being  made  to 
the  combat/weapon  systems  during  a  maintenance  availability  are  significant  and  the  risk 
those  changes  pose  to  the  readiness  of  the  ship.  The  purpose  of  this  process  is  to  assist 
decision-makers  in  determining  if  a  ship  should  conduct  Combat  Systems  Ship 
Qualification  Trials  (CSSQT)  post-availability.  The  process  uses  warfare  area  criticality 
and  type  of  change  to  determine  the  significance  of  the  change.  Risk  is  assessed  by 
determining  the  likelihood  the  changes  will  impact  maintenance  readiness,  operational 
readiness  and  safety.  The  final  result  of  the  process  is  a  matrix  that  uses  the  significance 
and  risk  likelihood  values  to  graphically  represent  the  overall  impact  the  availability  has 
on  the  ship.  Where  a  ship  is  plotted  on  the  matrix  will  determine  the  recommendation  for 
that  ship. 
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EXECUTIVE  SUMMARY 


This  thesis  develops  a  process  for  quantifying  the  impact  a  major  overhaul/modernization 
has  on  a  ship’s  overall  readiness  in  tenns  of  the  significance  of  changes  and  the 
likelihood  of  risk.  The  need  for  such  a  process  is  because  there  is  not  a  quantitative 
process  in  place  for  designating  a  ship  for  Combat  Systems  Ship  Qualification  Trials 
(CSSQT).  The  Naval  Surface  Warfare  Center  Port  Hueneme  Division  (NSWC  PHD) 
created  a  process  and  this  thesis  investigates  that  process  and  develops  a  methodology  to 
determine  the  impact  to  ship’s  readiness  that  builds  from  the  current  work.  The  result  is  a 
tool  for  determining  the  significance  of  the  maintenance  availability,  a  tool  for  assessing 
the  likelihood  of  risk,  and  a  process  for  presenting  the  results  in  a  manner  that  allows 
decision-makers  to  easily  determine  if  a  ship  should  be  designated  to  conduct  a  CSSQT 
based  on  the  overall  impact  to  ship’s  readiness. 

The  governing  instruction  for  CSSQT,  NAVSEAINST  9093. 1C,  states  that  a  ship 
will  be  designated  for  a  CSSQT  if  it  is  undergoing  a  significant  overhaul/modemization 
availability.  The  instruction  does  not  describe  what  constitutes  significant;  therefore,  the 
first  step  is  to  make  a  decision  about  designating  a  ship  for  CSSQT  is  to  determine  if  the 
availability  is  significant.  NSWC  PHD  designed  a  procedure  for  determining  significance 
and  called  it  the  CSSQT  Decision  Tool.  A  functional  decomposition  was  used  to  review 
the  tool  and  the  following  problems  were  discovered: 

•  The  name  of  the  tool  does  not  accurately  reflect  the  purpose  of  the  tool. 
This  is  a  minor  issue  but  it  is  important  that  the  name  of  the  tool  describes 
the  intended  purpose.  The  current  tool  is  called  the  CSSQT  Decision  Tool 
which  indicates  the  purpose  of  the  tool  is  to  make  a  decision.  The  review 
determined  that  the  actual  purpose  of  the  tool  is  to  assess  the  significance 
of  changes  being  made  to  the  combat/weapon  systems.  To  better  reflect 
the  purpose  of  the  tool  the  name  has  been  changed  to  Significance 
Assessment  tool 

•  There  is  an  inaccurate  accoun  tability  of  all  combat/weapon  systems.  Ten 
ships  were  assessed  using  NSWC  PHD’s  process  and  the  review  of  the 
assessments  found  a  majority  of  the  ships  had  an  inaccurate  accountability 
of  the  combat/weapon  systems  onboard  the  ship.  The  assessments  were 
missing  systems,  had  the  wrong  systems  listed  and  grouped  training 
systems  together.  The  training  systems  were  grouped  because  they  were 

xiii 


not  critical  to  any  warfare  area  and  therefore  any  changes  made  to  those 
systems  did  not  need  to  be  accounted  for.  Training  systems  are  a  major 
part  of  the  crew’s  readiness  and  need  to  be  assessed  individually  in  the 
same  manner  as  the  other  systems.  To  achieve  an  accurate  assessment, 
careful  attention  to  include  the  correct  systems  must  occur. 

•  The  significance  calculation  result  is  ambiguous.  The  result  of  the  process 
is  a  percentage,  but  it  is  unclear  what  the  value  is  a  percentage  of.  The  best 
approach  for  showing  significance  is  to  use  an  interval  scale  where  a 
significance  of  10  is  extremely  significant  and  a  significance  of  0  is  not 
significant. 

•  The  significance  calculation  does  not  account  for  non-critical  systems. 
Systems  not  critical  to  a  warfare  area  are  not  accounted  for  in  the 
assessment,  even  if  the  system  is  being  altered.  CSSQT  validates  and 
verifies  all  combat/weapon  systems,  so  it  is  important  every  system  is 
included  in  the  assessment.  A  weight  is  given  to  non-critical  systems  and 
added  to  the  function  to  determine  significance. 

•  A  completed  assessment  can  result  in  a  spreadsheet  that  contains  empty 
cells  and  cells  that  contain  FALSE  as  an  entry.  Systems  not  being  changed 
are  left  blank  on  the  assessment  spreadsheet  which  leads  to  invalid 
calculations  and  cells  that  contain  FALSE.  This  practice  makes  the 
assessment  look  incomplete.  To  solve  this,  a  category  was  added  to 
indicate  that  a  system  is  not  being  changed  during  the  availability 

The  likelihood  of  risk  to  a  ship’s  readiness  can  be  used  as  an  indicator  of  how  the 
maintenance  availability  impacts  a  ship.  This  thesis  developed  a  method  for  determining 
the  likelihood  a  ship’s  readiness  will  be  impacted  by  the  changes  being  made  to  the 
combat/weapon  systems.  The  risk  assessment  tool  developed  by  NSWC  PHD  was 
compared  to  the  new  methodology  and  the  following  issues  were  discovered: 

•  The  risk  areas  and  risk  factors  do  not  align  with  the  objectives  outlined  in 
NAVSEAINST  9093. 1C.  The  risk  areas  proposed  by  NSWC  PHD  are 
Effectiveness,  Readiness,  Safety  and  Personnel  but  the  instruction 
specifically  states  that  the  objectives  of  CSSQT  are:  “To  ensure  a  ship’s 
combat  system  has  a  high  level  of  operational  readiness,  maintenance 
readiness,  and  is  safe”  (NAVSEA  2006).  The  instruction  also  outlines  the 
factors  that  are  assessed  to  ensure  the  objectives  are  met.  Aligning  the  risk 
assessment  with  the  instruction  ensures  the  assessment  fully  captures  the 
intent  of  CSSQT. 

•  A  large  number  of  risk  factors  required  subjective  input  from  outside 
sources.  The  risk  assessment  is  intended  to  be  quantitative,  not  qualitative, 
and  the  proposed  process  uses  subjective  assessments  to  determine  risk. 
The  assessment  must  be  consistent  for  each  ship  being  evaluated  to  ensure 
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the  results  are  consistent.  Relying  on  subjective  assessment  can  lead  to 
inconsistencies  and  can  also  make  completing  the  assessment  difficult.  All 
subjectivity  has  been  removed  from  the  risk  assessment  and  it  relies  only 
on  data  that  can  be  found  in  on-line  databases. 

•  The  output  of  the  assessment  is  ambiguous.  The  result  of  the  assessment  is 
in  the  fonn  of  a  percentage,  but  it  is  unclear  what  this  percentage 
represents.  The  same  scale  that  is  used  in  the  Significance  Assessment  will 
be  used  for  the  risk  assessment,  where  10  is  extremely  likely  and  0  is  not 
likely. 

The  final  result  is  a  process  that  uses  the  significance  assessment  and  risk 
assessment  to  determine  the  overall  impact  to  a  ship’s  readiness.  To  accomplish  this,  a 
central  spreadsheet  for  the  results  of  the  assessments  was  created.  This  is  called  the 
CSSQT  Determination  Dashboard.  It  collects  the  results  from  the  assessments  and 
determines  a  score  for  the  significance  and  risk  likelihood.  These  scores  are  plotted  on 
the  Detennination  Matrix,  shown  in  Figure  1,  for  each  ship  being  assessed.  The  plot 
provides  a  graphical  view  of  the  results  and  the  quadrants  of  the  matrix  are  labeled  to 
give  a  recommendation  based  on  the  significance  and  likelihood. 


FY15  CSSQT  Determination  Matrix 
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I.  INTRODUCTION 


This  thesis  will  investigate  and  detennine  a  quantitative  procedure  to  assess  the 
overall  impact  to  a  ship’s  operational  readiness  after  a  major  modernization.  The 
procedure  is  required  as  part  of  a  new  systematic  process  for  designating  a  ship  to 
conduct  Combat  System  Ship  Qualification  Trials  (CSSQT).  Naval  Sea  Systems 
Command,  Port  Hueneme  Division  (NSWC  PHD)  detennined  that  there  is  a  “ Lack  of  an 
objective  and  validated  process  to  determine  CSSQTsf  there  is  an  “ Inability  to  predict 
risk  for  post  availability  impacts  3  to  4  years  in  advance, ”  and  there  are  “ Multiple 
interpretations  of  ‘significant’”  (B.  Hazel  and  S.  Matthews,  unpublished  Powerpoint 
slide).  In  2012,  NSWC  PHD  introduced  a  new  process  for  determining  which  ships 
should  be  designated  for  CSSQT  and  the  process  includes  a  tool  for  assessing  the 
significance  and  a  tool  for  assessing  the  risk.  This  thesis  will  present  a  methodology  to 
review  the  two  tools  and  present  a  new  procedure  for  assessing  overall  impact  to  a  ship’s 
readiness. 

A.  PROBLEM  STATEMENT 

According  to  NAVSEAINST  9093. 1C,  “CSSQTs  will  be  scheduled  for  all  surface 
combatants,  aircraft  carriers,  and  amphibious  ships  completing  new  construction  or 
significant  conversion/modernization  availabilities”  (2006).  The  problem  is  how  to  define 
a  significant  conversion/modemization.  Prior  to  2012,  it  was  the  responsibility  of  the 
“applicable  Program  Executive  Offices  (PEOs)/Program  Managers  (PMs)  in  coordination 
with  the  Type  Commanders,  CSSQT  Certification  Authority  and  Naval  Sea  Systems 
Command  (SEA  62)  to  designate  a  ship  for  CSSQT  post-availability”  (NAVSEA  2006) 
based  on  subjective  assessment  and  the  availability  of  funding.  NSWC  PHD  (CSSQT 
Certification  Authority)  has  determined  the  need  for  a  process  that  removes  subjectivity 
and  relies  on  a  quantitative  assessment  to  determine  the  significance  of  combat/weapon 
system  changes  made  during  a  maintenance  availability  in  order  to  make  a  decision  if  a 
CSSQT  is  warranted  for  a  specific  ship.  Furthermore,  NSWC  PHD  deemed  it  necessary 
to  assess  the  risk  associated  with  the  significance  of  the  conversion/modemization.  An 
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assessment  of  the  significance  and  the  risk  to  the  ship  will  give  decision-makers  the 
information  to  fully  understand  the  nature  of  the  modernization  and  make  a  better 
decision  when  designating  a  ship  for  CSSQT.  A  procedure  that  can  be  applied  to  all  ships 
scheduled  for  a  conversion/modernization  needs  to  be  designed  that  will  assess  the 
significance  and  risk  the  conversion/modernization  will  have  on  the  ship’s  readiness. 

B.  SCOPE 

CSSQT  is  performed  on  every  class  of  surface  combatant  ships  and  it  is  necessary 
to  assess  each  ship  undergoing  a  conversion/modernization,  but  this  research  will  restrict 
the  assessment  to  amphibious  ships  and  aircraft  carriers.  The  research  will  analyze  ships 
conducting  a  maintenance  availability  in  fiscal  year  2015.  The  focus  will  be  on  reviewing 
the  tools  created  by  NSWC  PHD  and  developing  an  objective  process  for  determining  the 
significance  and  risk  associated  with  a  major  conversion/modernization. 

C.  ORGANIZATION 

The  next  chapter  will  discuss  the  background  of  CSSQT  and  present  the  current 
work  being  done  by  NSWC  PHD.  The  third  chapter  describes  the  methodology  used  to 
review  the  current  work  and  the  process  used  to  develop  the  new  risk  and  significance 
tools.  The  fourth  chapter  will  present  the  results  of  the  methodology.  The  final  chapter 
will  discuss  conclusions  and  recommendations  for  further  research. 
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II.  BACKGROUND 


A.  COMBAT  SYSTEMS  SHIP  QUALIFICATION  TRIALS 

CSSQT  is  a  major  evolution  that  requires  years  of  planning  and  two  to  three 
months  to  execute  depending  on  the  complexity  of  the  combat  systems  on  the  individual 
ship.  The  execution  of  a  CSSQT  is  governed  by  NAVSEA  Instruction  9093. 1C,  Combat 
System  Ship  Qualification  Trials  for  Surface  Ships.  The  instruction  outlines  the  purpose, 
policy,  objectives,  definitions,  guidelines  and  the  planning  process  for  any  surface  ship 
designated  to  conduct  a  CSSQT.  Purpose  and  policy  are  important  to  understanding  why 
CSSQTs  are  conducted  and  what  is  required  to  designate  a  ship  to  perfonn  one.  They  also 
provide  valuable  information  on  how  to  approach  creating  a  process  for  making  a 
recommendation  a  ship  conduct  a  CSSQT.  The  purpose  of  CSSQT  is: 

To  verify  and  validate  that  an  individual  ship’s  combat/weapon  systems 
have  been  installed  correctly  and  can  be  operated  and  maintained  in  a  safe 
and  effective  manner.  This  is  accomplished  by  assisting  ship’s  force  in 
achieving  (1)  a  sustainable  level  of  combat/weapon  system  operational 
readiness  and  (2)  a  maintainable  level  of  material  readiness.  (NAVSEA 
2006) 

The  purpose  indicates  after  a  successful  completion  of  CSSQT  the  ship  will  be 
able  to  operate  and  maintain  the  combat/weapon  systems  at  a  higher  level  than  if  it  had 
not  conducted  a  CSSQT.  It  can  be  inferred  a  ship  should  conduct  a  CSSQT  if  there  are 
circumstance  that  reduce  the  ship’s  ability  to  operate  and  maintain  the  combat/weapon 
systems.  The  policy  of  CSSQT  provides  the  circumstances  and  gives  guidance  to  when  a 
ship  should  conduct  a  CSSQT. 

CSSQTs  will  be  scheduled  for  all  surface  combatants,  aircraft  carriers,  and 
amphibious  ships  completing  new  construction  or  significant 
conversion/modernization  availabilities.  Designation  of  post  availability 
ships  for  CSSQTs  will  be  made  by  applicable  Program  Executive  Offices 
(PEOs)/Program  Managers  (PMs)  in  coordination  with  the  Type 
Commanders,  CSSQT  Certification  Authority  and  Naval  Sea  Systems 
Command  (SEA  62).  These  designations  will  be  based  on  the  extent  of 
combat  system  changes  made  during  the  availability  or  the  cumulative 
effect  of  combat/weapon  systems  changes  since  the  last  CSSQT. 
Determinations  to  conduct  CSSQTs  and  their  scope  will  be  developed  by  a 
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collaborative  system  engineering  process.  Following  CSSQT,  each 

applicable  mission  area  will  be  qualified  to  be  safe  and  effective  based  on 

combat/weapon  system  performance  and  observed  crew  proficiency. 

(NAVSEA  2006) 

A  ship  is  designated  to  execute  a  CSSQT  at  an  annual  review  board  conducted  by 
SEA  62.  The  board  looks  at  all  surface  ships  conducting  a  maintenance  availability  in  the 
next  four  years  and  detennines  if  the  ship  requires  a  CSSQT.  Recommendations  from 
PEOs,  PMs,  TYCOMs  and  CSSQT  Execution  Agency,  type  of  availability  and  the 
number  of  years  since  the  ship  conducted  its  last  CSSQT  allow  the  review  board  to  make 
a  decision  if  a  ship  should  conduct  a  CSSQT.  Once  a  ship  is  designated  to  conduct  the 
trials,  a  test  plan  will  be  developed  based  on  combat  system  requirements.  The  test  plan  is 
ship  specific  in  which  the  scope  is  unique,  but  the  objectives  are  the  same. 

The  scope  of  a  CSSQT  is  based  on  the  warfare  areas  of  the  ship.  The  warfare 
areas  that  can  be  included  in  a  CSSQT  are: 

•  Air  Defense  Warfare  (ADW) 

•  Strike  Warfare  (STW) 

•  Surface  Warfare  (SUW) 

•  Undersea  Warfare  (USW) 

•  Expeditionary  Warfare  (EXW) 

•  Electronic  Warfare  (EW) 

CSSQT  assess  all  combat/weapon  systems  that  support  each  warfare  area.  A 
combat/weapon  system  is  defined  as  a  functional  grouping  of  all  the  shipboard 
equipment,  computer  programs,  personnel  and  documentation,  plus  other  critical  systems 
that  are  designed  to  detect,  track,  identify,  communicate,  process,  evaluate,  and  execute 
the  engagement  of  enemy  forces,  either  actively  passively  (NAVSEA  2006).  Not  all  ships 
are  capable  of  all  the  above  listed  warfare  areas  so  the  scope  of  each  CSSQT  will  depend 
on  the  capabilities  of  the  ship.  Regardless  of  the  scope  of  the  CSSQT,  the  objectives 
remain  constant. 

The  objectives  of  CSSQT  are  “to  verify  and  validate  that  the  combat/weapon 
systems  have  been  installed  correctly  and  can  be  operated  and  maintained  in  a  safe  and 

effective  manner”  (NAVSEA  2006).  Verification  and  validation  is  accomplished  by 
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subject  matter  experts  (SMEs)  assisting  the  crew  with  Planned  Maintenance  System 
(PMS)  actions  and  also  conducting  in-port  and  at-sea  combat  system  exercises.  The 
exercises  use  live  and/or  simulated  environments  to  stress  the  combat  systems  and  ship’s 
force  in  order  to  determine  the  level  of  maintenance  readiness  and  operational  readiness. 
Overall  readiness  is  assessed  using  Measures  of  Effectiveness  (MOEs)  for  adequacy  of 
system  documentation,  logistics  support,  maintenance,  and  operational  readiness. 
Assessing  the  MOEs  and  meeting  the  objectives  of  the  CSSQT  ensure  the  ship  has  a  safe 
and  effective  combat  system  and  is  prepared  for  future  training  and  operations. 

CSSQT  is  a  valuable  experience  for  the  ship’s  force  because  of  the  opportunity  to 
work  closely  with  subject  matter  experts  (SMEs)  while  the  crew  leams  to  maintain  the 
equipment  and  experiences  how  to  operate  it  in  real-world  situations.  The  other  benefit  of 
CSSQT  is  the  opportunity  for  engineers  to  conduct  integrated  Developmental  and 
Operational  Testing  (DT/OT),  to  perform  tactics  validation  and  also  conduct  fleet 
training.  The  problem  is  a  ship  may  only  conduct  one  or  two  CSSQTs  during  the  entire 
life  of  the  ship,  but  combat  and  weapons  systems  are  continuously  being  upgraded  and/or 
added  through  the  Navy  Modernization  Plan  (NMP).  It  would  be  ideal  for  a  ship  to 
conduct  a  CSSQT  after  every  major  modernization,  but  due  to  the  cost  to  conduct  one  it 
is  not  feasible.  Based  on  the  assessment  of  risk  and  significance  it  would  be  possible  to 
recommend  a  CSSQT  that  is  scoped  down  to  ensure  the  major  changes  to  the 
combat/weapon  systems  are  validated  and  verified. 

B.  CURRENT  WORK 

NSWC  PHD  created  the  CSSQT  Determination  Process  in  2012  to  capture  the 
process  outlined  in  NAVSEAINST  9093. 1C.  It  introduces  two  tools  for  determining  the 
significance  of  changes  being  made  to  the  ship  and  assessing  the  risk  associated  with  the 
changes.  Figure  2  shows  the  beginning  of  the  detennination  process.  The  full  flow  chart 
can  be  found  in  Appendix  A. 
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Figure  1.  CSSQT  Determination  Process  (From  B.  Hazle  and  S.  Matthews,  unpublished  Powerpoint  slide) 
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The  process  begins  prior  to  the  System  Engineering  Board  (SEB)  which  occurs 
60  days  before  the  CSSQT  Annual  Review  Board  (ARB)  (not  pictured  in  Figure  2).  The 
SEB  reviews  each  ship  that  is  scheduled  to  undergo  an  overhaul/modemization 
availability  in  the  next  four  years.  The  SEB  uses  the  assessments  from  the  CSSQT 
Decision  Tool  process  and  the  CSSQT  Risk  Assessment  process  to  make  a 
recommendation  to  the  sponsors/CSSQT  community  that  a  ship  should  or  should  not 
conduct  a  CSSQT.  Ships  reviewed  in  previous  years  are  reassessed  for  any  changes  to  the 
ship’s  availability.  The  Decision  Tool  and  Risk  Assessment  processes  allow  the  SEB  to 
make  a  more  informed  decision  about  designating  ships  for  CSSQT.  Each  process  will  be 
discussed  in  more  detail  in  the  following  sections. 

1.  CSSQT  Decision  Tool  Process 

The  purpose  of  the  CSSQT  Decision  Tool  process  is  to  detennine  the  significance 
of  the  overhaul/modernization  availability.  The  decision  tool  is  a  simple  spreadsheet 
which  lists  the  combat/weapon  systems  onboard  the  ship  and  assigns  weights  to  each 
system  based  on  the  criticality  to  each  warfare  area  and  the  type  of  modernization. 
Weights  used  for  criticality  and  type  of  modernization  were  determined  by  the  developers 
of  the  process.  A  detailed  analysis  of  how  the  weights  affect  the  results  will  be  discussed 
in  Chapter  III.  The  criticality  weight  is  multiplied  by  the  modernization  weight  to  yield  a 
warfare  area  qualification  weight.  The  sum  of  the  warfare  area  qualification  weights 
divided  by  the  sum  of  the  warfare  criticality  weights  produces  a  percentage  that 
represents  the  significance  of  changes  being  made  to  each  warfare  area.  The  average  of 
the  significance  values  represents  the  overall  significance  of  the  availability.  The 
spreadsheet  is  populated  using  Warfare  Interface  System  Engineering  (WISE)  database 
and  Ship  Change  Documents  (SCDs). 

NSWC  PHD  uses  the  WISE  database  to  determine  when  ships  will  be  entering 
into  a  modernization  period  and  to  detennine  which  systems  will  be  changed.  The 
database  contains  the  current  combat  and  weapon  systems  onboard  the  ship,  which 
software  version  the  systems  are  running  and  also  when  scheduled  upgrades  to  the 
systems  will  occur.  Each  availability  has  a  corresponding  Warfare  System  Interface 
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Diagram  (WSID)  that  is  assessable  from  the  WISE  database.  The  WSID  is  a  graphic  that 
shows  all  combat  and  weapon  systems,  how  they  are  connected  to  the  command  and 
control  system  and  which  systems  are  being  modified  during  the  availability.  It  also 
contains  a  list  of  the  applicable  SCDs  and  a  description  of  the  changes.  An  example  of  a 
WSID  can  be  found  in  Appendix  B.  The  WSID  is  used  to  populate  the  decision  tool 
spreadsheet  with  the  systems  and  get  the  applicable  SCD  numbers,  but  the  SCDs  are 
located  in  another  database. 

A  SCD  is  the  authorized  document  for  any  ship  change  during  maintenance 
availabilities.  All  SCDs  are  located  on  the  Navy  Data  Environment  Navy  Modernization 
Entitled  Process  database.  SCDs  contain  a  lot  of  information  but  the  information  used  by 
NSWC  PHD  is  about  the  type  of  modification  being  made  to  the  system.  It  indicates  if 
the  system  is  being  added  to  the  ship,  a  new  system  is  replacing  an  old  system,  the 
modification  is  adding  a  new  capability,  improving  capability,  improving  performance,  or 
improving  infrastructure.  This  information  is  used  in  the  decision  tool  to  determine  the 
modernization  weight  values.  The  weights  used  for  each  type  of  modification  are  shown 
in  Figure  3.  If  a  system  is  not  being  changed  it  receives  a  value  of  FALSE  because  there 
is  not  a  category  for  a  system  not  being  altered. 


Type 

Weight 

Replace  with  New 

l 

Add  Capability 

l 

Improve  Capability 

l 

Improve  Performance 

0.5 

Improve  Infrastructures 

0.25 

Figure  2.  Modification  Weights  (From  B.  Hazel  and  S.  Matthews, 

unpublished  Excel  spreadsheet) 

The  SCD  does  not  show  how  vital  a  system  is  to  a  specific  warfare  area.  The 
assessment  of  criticality  is  based  on  knowledge  of  how  each  system  functions  to  support 
a  specific  warfare  area.  For  example,  a  three-dimensional  air  search  radar  may  be  critical 
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to  anti-air  warfare  but  unnecessary  for  the  mission  of  surface  warfare  because  surface 
warfare  requires  a  surface  search  radar.  NSWC  PHD  uses  the  categories  and  weights 
shown  in  Figure  4  to  apply  criticality  to  each  system. 


Category 

Comments 

Weight 

Critical 

Without  this  system  cannot  perform  the 
mission  (Mission  Failure) 

l 

Necessary 

Without  this  system  mission  can  be 
performed  but  at  a  reduced  success  rate 
(Reduced  Performance) 

0.5 

Like  to  have 

Without  this  system  mission  can  be 
performed  but  with  reduced  awareness 
(Reduces  Situational  Awareness) 

0.25 

Unnecessary  for  Mission 

System  does  not  affect  mission  (No  Effect) 

0 

Figure  3.  Mission  Area  Criticality  Categories  and  Weights  (From  B.  Hazel 
and  S.  Matthews,  unpublished  Excel  spreadsheet) 

The  decision  tool  allows  users  to  select  the  criticality  category  and  modernization 
type  from  a  drop  down  list.  When  a  selection  is  made  the  weight  value  is  automatically 
filled  in  and  the  spreadsheet  will  update  the  significance  percentage.  If  the  value  is 
greater  than  25%  then  a  CSSQT  is  recommended  and  indicated  on  the  spreadsheet.  A 
completed  decision  tool  is  in  Appendix  C.  After  the  CSSQT  Decision  Tool  process  the 
next  step  is  to  conduct  a  risk  assessment. 

2.  Risk  Assessment  Tool  Process 

The  risk  assessment  tool  process  has  not  been  put  into  practice  like  the  Decision 
Tool.  It  is  still  under  development  but  NSWC  PHD  has  developed  a  potential  tool  that 
could  be  used  to  assess  the  risk  of  a  ship  undergoing  a  major  modernization.  The  tool 
assesses  the  risk  to  effectiveness,  personnel,  safety  and  readiness  through  a  combination 
of  subjective  evaluations  of  the  availability  by  Subject  Matter  Experts  (SMEs),  historical 
data  about  the  ship,  and  current  data  about  the  availability.  The  developers  of  the  tool, 
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Bob  Hazel  and  Sandra  Matthew,  from  NSWC  PHD  defined  the  four  areas  of  risk  in  the 
following  way: 


a.  Effectiveness 

The  combat  system  is  not  able  to  perform  its  mission.  Effectiveness  risk  is 
determined  by  the  magnitude  of  changes  impacting  the  combat/weapon  system 
functionality  (B.  Hazel  and  S.  Matthews,  unpublished  Excel  spreadsheet). 

b.  Personnel 

The  combat  systems  cannot  be  operated  effectively  by  the  crew.  Personnel 
risk  is  determined  by  the  magnitude  of  changes  impacting  the  crew’s  ability  to  operate 
and  maintain  systems  (B.  Hazel  and  S.  Matthews,  unpublished  Excel  spreadsheet). 

c.  Safety 

Personal  injury  or  death  and/or  equipment  damage.  Safety  risk  is  assessed 
by  the  issues  associated  with  a  ship’s  modification,  in  conjunction  with  historical  data,  to 
determine  the  impact  with  individual  or  aggregate  consequences  (B.  Hazel  and  S. 
Matthews,  unpublished  Excel  spreadsheet). 

d.  Readiness 

The  combat  system  is  not  available/ready  to  perform  the  mission. 
Readiness  risk  is  determined  by  the  magnitude  of  changes  impacting  the  combat/weapons 
systems  ability  to  perform  its  mission  (B.  Hazel  and  S.  Matthews,  unpublished  Excel 
spreadsheet). 

There  are  1 8  risk  factors  assigned  to  one  or  more  of  the  categories  that  are 
used  to  assess  the  risk  and  Figure  5  shows  how  each  risk  factor  is  assigned  to  each  risk 
category  (an  X  indicates  that  risk  factor  is  used  in  assessing  the  risk  category). 
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Effective 

Personnel 

<u 

>4- 

ro 

i/> 

Material  Readiness 

RISK  FACTORS 

X 

X 

Years  since  last  CSSQT 

X 

X 

Years  since  last  live  firing 

X 

X 

X 

X 

Complexity  Change  of  Combat  System 

X 

X 

Number  of  Changes  in  1 LS  Support  (Tech  Doc) 

X 

X 

Number  of  Changes  in  1  LS  Support  (Test  Equip) 

X 

X 

Number  of  Changes  in  1  LS  Support  (Spares) 

X 

X 

Number  of  Changes  in  1  LS  Support  (PMS) 

X 

X 

X 

Training  Program  Planned  or  Developed/Delivered  (AIT) 

X 

X 

X 

Training  Program  Planned  or  Developed/Delivered  (LBTS) 

X 

X 

X 

Planned  ATG  training 

X 

X 

X 

Certifications/Qualifications  1  mpacted 

X 

X 

Number  of  open  R1  Trouble  Reports  (TRs) 

X 

X 

Ship's  force  ability  to  perform  the  mission 

X 

X 

Known  System  1  ncidents  of  risks 

X 

Known  Safety  Incidents  of  risks 

X 

Manpower  Requirements  adequately  reflected 

X 

X 

X 

Project  Officer/Project  Engineer  Evaluation  of  Risk 

X 

X 

X 

Baseline  Manager/Equivalent  (i.e.  Test  Con. /Dir.  Evaluation  of  Risk 

Figure  4.  Risk  Factor  Assignments  (From  B.  Hazel  and  S.  Matthews, 

unpublished  Excel  spreadsheet) 


The  risk  factors  and  how  they  are  assessed  is  shown  in  Figure  6. 
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RISK  FACTORS 

Assessment 

Years  since  last  CSSQT 

Enter  number  of  years  since  last  CSSQT 

Years  since  last  live  firing 

Enter  number  of  years  since  lastf  ring  (reduces) 

Complexity  Change  of  Combat  System 

Modernization  Impact  percentage  taken  from  the  Decision  tool 

Number  of  Changes  in  1 LS  Support  (Tech  Doc) 

PUBSAT:  Calculate  using  SCD's.  Count  number  of  SCDs  identified  as  impacting 
TECH  DOC  (Counting  multiple  SCDs  to  a  single  system  only  once)(X)/Total 
number  of  ships  systems  (XX)  Tech  docs  checked/tolal  number  of  systems 

Number  of  Changes  in  1  LS  Support  (Test  Equip) 

TESAT:  Calculate  using  SCD's.  CountnumberofSCDs  identified  as  impacting 
TEST  EQUIP  (Counting  multiple  SCDs  to  a  single  system  only  once )(X)/Total 
number  of  ships  systems  (XX)ech  docs  checked/total  number  of  systems 

Number  of  Changes  in  1  LS  Support  (Spares) 

LOGSAT:  Calculate  using  SCD's.  CountnumberofSCDs  identified  as  impacting 
SPARES  (Counting  multiple  SCDs  to  a  single  system  only  once)(X)/Total  number 
of  ships  systems  (XX)  ech  docs  checked/total  number  of  systems 

Number  of  Changes  in  1  LS  Support  (PMS) 

PUBSAT:  Calculate  using  SCD's.  CountnumberofSCDs  identified  as  impacting 
PMS  (Counting  multiple  SCDs  to  a  single  system  onlyonce)(X)/Total  number  of 
ships  systems  (XX)  ech  docs  checked/tobl  number  of  systems 

Training  Program  Planned  or  Developed/Delivered  (AIT) 

Modernization  Rep/SME 

Training  Program  Planned  or  Developed/Delivered  (LBTS) 

B/L  Mgr/SME 

Planned  ATG  training 

Training  SME 

Certifications/Qualifications  1  mpacted 

SCD  Section  15/23 

Number  of  open  R1  Trouble  Reports  (TRs) 

NWSCP  R1  definition 

Ship's  force  ability  to  perform  the  mission 

NWSCP-Aggregate  Assessment  of  number  of  Tactics,  Techniques,  and 
Procedures  (TTP),  workarounds,  limitations,  and  restrictions  indicates  ship’s  force 
can  perform  the  mission  PRI  3/4's) 

Known  System  Incidents  of  risks 

Modernization  Division  System  Engineer's  evaluate  known  system  incidents  of 
risks 

Known  Safety  Incidents  of  risks 

SME  evaluate  known  safety  incidents  of  risks 

Manpower  Requirements  adequately  reflected 

Fit/Fill 

Project  Officer/Project  Engineer  Evaluation  of  Risk 

Evaluation  of  Ship  Risk  introduced  from  the  specific  changes  occurring/occurred 
to  a  specific  ship  at  a  specific  Ship  yard  with  a  specific  Training  Team 

Baseline  Manager/Equivalent  (i.e.  Test  Con./Dir.  Evaluation  of  Risk 

Evaluation  of  Baseline  Risks  introduced  from  the  specific  changes 
occurring/occurred  to  a  specific  ship 

Figure  5.  Risk  Factors  and  How  they  are  Assessed  (From  B.  Hazel  and 
S.  Matthews,  unpublished  Excel  spreadsheet) 


To  complete  the  risk  assessment  each  risk  factor  is  assigned  a  weight 
based  on  the  assessments  that  were  conducted.  Figures  7  through  10  show  criteria  and 
weights  for  each  risk  factor. 
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Figure  6.  Effectiveness  Criteria  and  Weights  (From  B.  Hazel  and 
S.  Matthews,  unpublished  Excel  spreadsheet) 


Figure  7.  Personnel  Criteria  and  Weights  (From  B.  Hazel  and 
S.  Matthews,  unpublished  Excel  spreadsheet) 
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Risk  Factors 

High  1.0 

Medium  .5 

Low  .25 

Not  Effected  0 

Complexity  Change  of  Combat 

49%-25% 

24%-i% 

0% 

Certifications/Qualifications 

3-2 

1 

0 

Known  Safety  Incidents  of  risks 

Medium 

Low 

None 

Project  Officer/Project  Engineer 

Medium 

Low 

None 

Baseline  Manager/Equivalent  (i.e. 

T est  Con  ./Dir.  Evaluation  of  Risk 

Medium 

Low 

None 

Figure  8.  Safety  Criteria  and  Weights  (From  B.  Hazel  and  S.  Matthews, 

unpublished  Excel  spreadsheet) 


Figure  9.  Readiness  Criteria  and  Weights  (From  B.  Hazel  and  S.  Matthews, 

unpublished  Excel  spreadsheet) 


After  the  assessment  is  completed  the  results  are  entered  into  a 
spreadsheet  that  calculates  the  risk  score  for  each  risk  category  and  the  overall  risk  score 
for  the  ship.  An  example  of  a  completed  risk  assessment  is  shown  in  Figure  11. 
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ENTER  SHIP  NAME  AND  FY  OF  CSSQT 

Risk  Assessment 

Risk  Factors 

ENTER  DATA 

Effective 

Personnel 

Safe 

Material 

Readiness 

Years  since  last  CSSQT 

2 

0 

0 

Years  since  last  live  firing 

0.0 

Complexity  Change  of  Combat  System 

.5 

.5 

.5 

.5 

Number  of  Changes  in  ILS  Support  (Tech  Doc) 

19% 

.25 

.25 

Number  of  Changes  in  ILS  Support  (Test  Equip) 

52% 

1.0 

1.0 

Number  of  Changes  in  ILS  Support  (Spares) 

4% 

.25 

.25 

Number  of  Changes  in  ILS  Support  (PMS) 

4% 

.25 

.25 

Training  Program  Planned  or  Developed/Delivered 
(AIT)  ' 

Not  Planned 

1.0 

1.0 

1.0 

Training  Program  Planned  or  Developed/Delivered 
(LBTS) 

Planned 

.25 

.25 

.25 

Planned  ATG  training 

Planned 

.25 

.25 

.25 

Certifications/Qualifications  Impacted 

13 

.5 

.5 

.5 

Number  of  open  R1  Trouble  Reports  (TRs) 

5 

1.0 

1.0 

Ship’s  force  ability  to  perform  the  mission 

15 

1.0 

1.0 

Known  System  Incidents  of  risks 

Medium 

.5 

.5 

Known  Safety  Incidents  of  risks 

Medium 

.5 

Manpower  Requirements  adequately  reflected 

Yes 

.25 

Project  Officer/Project  Engineer  Evaluation  of  Risk 

Low 

.25 

.25 

.25 

Baseline  Manager/Equivalent  (i.e.  TestCon./Dir. 
Evaluation  of  Risk 

Low 

.25 

.25 

.25 

TOTAL 

44.64% 

45.45% 

50.00% 

40.00% 

38.64% 

Figure  10.  Completed  Risk  Assessment  Example  (From  B.  Hazel  and 
S.  Matthews,  unpublished  Excel  spreadsheet) 


The  risk  score  for  each  category  is  the  sum  of  risk  factor  weights  divided 
by  the  total  number  of  risk  factors  for  the  specific  category.  The  total  score  for  the  overall 
risk  assessment  is  the  sum  of  all  risk  factor  weights  divided  by  the  total  number  of  risk 
factors.  Each  risk  category  is  given  a  severity  rating  based  on  the  category  score: 

•  >51%  Red  Severity  Rating 

•  >26%  and  <51%  Yellow  Severity  Rating 

•  >1%  and  <26%  Green  Severity  Rating 

•  0%  and  <1%  Blue  Severity  Rating 

The  higher  the  score  the  greater  risk  the  ship  may  experience  post¬ 
availability  to  effectiveness,  personnel,  safety  and  readiness.  The  risk  assessment  gives 
the  CSSQT  Certification  Authority  and  other  decision-makers  the  ability  to  make  a 
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stronger  recommendation  for  a  ship  to  be  designated  to  conduct  a  CSSQT.  A  low  risk 
assessment  score  indicates  the  changes  being  made  to  the  ship  will  not  have  an  impact  on 
the  readiness  of  the  ship  and  therefore  a  CSSQT  may  not  be  required  even  if  the  CSSQT 
decision  tool  recommends  a  CSSQT. 

The  next  chapter  will  investigate  the  current  work  to  find  if  there  is  a 
better  approach  to  the  two  tools.  The  tools  are  undeveloped  and  lack  validation  due  to  the 
short  time  they  have  been  use.  The  goal  is  to  develop  a  tool  that  captures  both  the 
significance  and  risk  in  one  tool/process  rather  than  having  two  separate  tools  that  are  not 
coupled. 
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III.  METHODOLOGY 


The  approach  used  for  this  research  consists  of  three  stages.  The  first  stage 
develops  a  methodology  for  reviewing  the  CSSQT  Decision  Tool  process  and  detennines 
if  it  is  achieving  the  desired  results  of  the  process.  The  second  stage  creates  a 
methodology  for  assessing  risk  and  reviews  the  Risk  Assessment  process  and.  The  third 
stage  develops  an  approach  to  present  significance  and  risk  in  a  manner  that  gives 
decision-makers  a  better  understanding  of  how  the  modernizations  are  impacting  the 
ships  in  a  given  fiscal  year.  This  chapter  will  only  present  the  methodologies  for  each 
stage.  The  results  of  the  methodologies  will  be  presented  in  Chapter  IV. 

A.  CSSQT  DECISION  TOOL  REVIEW 

The  goal  of  this  stage  is  to  create  a  methodology  to  review  the  current  CSSQT 
Decision  Tool  process  being  used  by  NSWC  PHD.  The  methodology  needs  to  determine 
what  the  process  is  trying  to  accomplish  and  what  is  needed  to  do  this.  Another  way  to 
say  this  would  be  what  is  the  function  of  the  process  and  what  functions  are  required  to 
achieve  the  overall  function.  When  looked  at  in  this  way  the  best  approach  is  to  use  a 
functional  decomposition. 

A  functional  decomposition  will  show  what  functions  the  tool  will  need  to  do  to 
achieve  the  overall  purpose.  The  first  step  is  to  define  the  overall  function  of  the  process. 
The  second  step  is  to  create  a  functional  decomposition  and  the  third  step  is  to  examine 
the  current  tool  to  see  if  it  performs  the  functions.  The  Decision  Tool  review  will  use  a 
completed  assessment  of  10  ships  conducting  modernization  availabilities  in  fiscal  year 
2015. 

The  purpose  of  the  CSSQT  Decision  Tool  process  is  to  determine  if  the 
modifications  being  made  to  the  ship  during  a  maintenance  availability  are  significant. 
The  need  for  this  determination  is  because  NAVSEAINST  9093. 1C  states  that  ships  will 
conduct  a  CSSQT  if  the  overhaul/modernization  is  significant.  Defining  significant  is 
essential  to  the  assessment.  Significant,  for  the  purpose  of  this  process,  is  defined  as  any 
change  or  combination  of  changes  that  reduce  the  overall  combat  readiness  of  a  ship. 
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Using  the  definition,  a  maintenance  availability  is  not  significant  if  the  changes  being 
made  to  the  combat/weapon  systems  do  not  reduce  the  ship’s  ability  to  execute  its 
assigned  missions.  However,  there  can  be  degrees  of  how  significant  the  changes  are 
beyond  not  significant.  If  the  impact  on  the  combat  readiness  is  minimal  then  it  can  be 
determined  that  the  overhaul  is  somewhat  significant.  When  the  changes  greatly  reduce 
the  combat  readiness  than  the  overhaul  is  extremely  significant.  To  determine  just  how 
significant  the  changes  are,  a  scale  needs  to  be  implemented  to  quantify  the  changes 
being  made  to  the  ship. 

For  the  CSSQT  Decision  Tool  to  accurately  determine  how  significant  the 
overhaul  is,  it  must  first  define  a  quantifiable  scale  for  significance.  Significance  is  the 
quantitative  value  for  how  significant  the  modifications  the  overhaul  is;  therefore,  a  finite 
interval  scale  can  be  used  to  assign  a  value  to  how  significant  the  changes  are.  A  scale 
that  spans  from  0  to  10,  where  0  is  not  significant  and  10  is  extremely  significant  will 
allow  the  process  to  assign  values  for  significance.  The  overall  function  of  the  CSSQT 
Decision  Tool  is  to  quantify  the  impact  to  each  system  in  terms  of  significance  so  the 
aggregate  of  all  the  systems  yields  a  value  that  can  be  used  on  an  interval  scale  to 
determine  if  the  overhaul  is  significant. 

The  approach  for  determining  the  significance  of  changes  requires  quantifying  the 
impact  the  changes  have  on  the  combat  system.  The  functions  of  the  process  need  to 
provide  a  quantitative  analysis  of  the  changes  being  made  to  the  different  systems  based 
on  their  importance  to  the  warfare  areas  and  the  type  of  modification  being  made  to  the 
systems.  The  result  of  this  analysis  will  yield  the  level  of  change  being  made  to  the 
combat/weapon  system  and  the  level  can  be  used  on  an  interval  scale  to  show  if  the 
change  is  significant.  The  process  will  have  top-level  functions  and  sub-functions  that  are 
required  to  ensure  the  overall  function  is  possible.  These  functions  are  shown  in  the 
functional  decomposition  in  Figure  12.  The  following  sections  describe  each  function  and 
reviews  of  the  CSSQT  Decision  Tool’s  ability  to  perform  the  functions. 
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Figure  11.  Significance  Functional  Decomposition 

1.  Account  for  All  Systems 

The  first  step  in  determining  significance  is  to  accurately  account  for  all 
combat/weapon  systems  installed  on  the  ship.  Ensuring  all  systems  are  included  in  the 
assessment  will  reduce  the  risk  of  omitting  a  major  change  that  could  potentially  impact 
the  ship’s  readiness.  The  current  process  does  list  the  systems  but  the  review  found  the 
list  was  inaccurate  and  also  grouped  systems  based  on  common  functions.  The  list  of 
systems  in  the  reviewed  completed  assessment  contained  systems  that  were  not  installed 
and  also  did  not  list  installed  systems.  Training  systems  were  grouped  into  one  system 
which  resulted  in  a  lack  of  understanding  which  system  is  being  modified,  in  the  case 
there  is  a  modification.  In  most  cases,  at  least  one  of  the  three  training  systems  onboard  a 
ship  were  being  modified  in  some  way;  but,  it  was  unclear  which  system  was  being 
affected.  Training  systems  are  considered  non-critical  to  the  mission  of  the  ship; 
however,  they  are  vital  to  ensuring  the  crew  is  trained  to  execute  the  mission.  They  must 
be  treated  in  the  same  manner  as  other  major  systems  and  each  system  evaluated 
individually  versus  grouped. 

The  class  of  ship  may  be  the  same  but  the  systems  onboard  the  ship  can  differ 
immensely.  It  is  vital  to  the  process  that  each  system  is  accounted  for  so  changes  are 
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assessed  properly.  The  WISE  database  has  the  capability  to  allow  users  to  download  a  list 
of  systems,  in  spreadsheet  format,  onboard  a  ship  post-availability.  This  capability  must 
be  used  for  each  ship  being  assessed  to  ensure  all  systems  are  accounted  for.  The  next 
function  is  to  detennine  how  critical  each  system  is  to  each  warfare  area. 

2.  Determine  System  Criticality 

System  criticality  is  dependent  on  how  the  system  integrates  into  the  detect-to- 
engage  path  for  each  warfare  area.  The  assessment  is  done  by  determining  if  the  warfare 
area  mission  can  still  be  accomplished  without  that  specific  system.  The  criticality  will 
determine  the  weight  that  will  be  assigned  to  the  system.  The  current  process  does  apply 
mission  area  criticality  to  each  system  and  the  categories  were  discussed  in  Chapter  II. 
The  review  found  that  the  first  three  categories  are  appropriate  for  assessing  the  criticality 
but  the  Unnecessary  for  Mission  category  needs  to  be  addressed.  All  systems  onboard 
ships  have  some  necessity  in  completing  the  mission.  For  this  reason,  the  category  needs 
to  be  changed  to  Non-Critical.  This  change  makes  it  clear  that  a  system  may  not  directly 
be  required  for  a  mission  area  but  is  still  important  to  the  overall  mission  readiness  of  the 
ship. 


a.  Apply  Criticality  Weights 

The  purpose  of  a  criticality  weight  is  to  quantify  the  criticality.  A  system 
is  assigned  a  category  based  on  how  critical  it  is  to  the  warfare  area,  but  the  category 
needs  to  be  converted  to  a  number  for  use  in  the  fonnula  for  determining  impact  on  the 
warfare  area.  The  scale  used  for  criticality  is  a  non-dichotomous  ordinal  scale  because  it 
spans  a  spectrum  of  how  critical  a  system  is  to  a  warfare  area.  Typically,  a  scale  of  this 
nature  will  have  numbers  associated  with  each  category.  For  example,  a  survey  of 
opinion  might  have  the  categories  of  completely  agree,  mostly  agree,  no  opinion,  mostly 
disagree,  and  completely  disagree  where  the  values  for  the  categories  would  be  5,  4,  3,  2 
and  1,  respectfully.  The  equal  separation  between  values  demonstrates  a  consistent 
decline  if  the  magnitude  of  the  categories.  If  the  separation  is  not  consistent  in  this  type 
of  scale  than  the  concern  is  the  full  spectrum  is  not  captured  by  the  categories. 
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Capturing  the  full  spectrum  is  important  for  determining  the  criticality 
weights.  If  a  system  is  not  critical,  the  next  category  should  be  indicative  of  the  next  level 
of  criticality.  The  value  should  also  represent  one  step  down  from  the  previous  value.  The 
range  between  the  values  can  be  considered  the  magnitude  of  difference  between 
categories.  When  the  separation  between  category  values  is  equal,  there  is  an  equal 
magnitude  of  difference  between  categories  indicating  a  consistent  decrease  from  one 
category  to  the  next.  If  the  magnitude  of  difference  is  not  equal,  it  could  indicate  another 
category  is  required  to  ensure  the  scale  includes  all  required  categories.  The  criticality 
categories,  Critical,  Necessary,  Like  to  Have,  and  Non-Critical  can  be  plotted  on  a 
number  line,  with  equal  intervals,  that  ranges  from  0  to  1,  and  a  system  categorized  as 
Critical  would  earn  a  value  of  1  and  a  Non-Critical  system  would  earn  a  value  of  0.  A 
Like  to  Have  system  would  have  a  value  of  0.33  and  a  Necessary  system  would  have  a 
value  of  0.66.  The  equal  separation  ensures  the  categories  represent  the  entire  range  of 
criticality. 

NSWC  PHD  gives  a  Critical  system  a  weight  of  1,  a  Necessary  system  a 
weight  of  0.50  and  a  Like  to  Have  system  a  weight  of  0.25  indicating  a  Critical  system  is 
two  magnitudes  more  critical  as  a  Necessary  system,  but  a  Necessary  systems  is  only  a 
single  magnitude  more  critical  than  a  Like  to  Have  system.  The  problem  with  this 
approach  is  there  is  no  justification  for  why  the  separation  between  categories  is  different. 
It  should  be  understood  why  a  Critical  system  is  more  critical  than  a  Necessary  system, 
but  NSWC  PHD  does  not  address  the  reason  in  the  description  of  the  tool.  It  is  possible 
that  the  selection  of  weights  does  not  affect  the  results  of  the  analysis  and  therefore  a 
justification  of  why  the  values  used  for  the  weights  is  not  necessary. 

A  sensitivity  analysis  on  the  values  for  the  weights  was  conducted  to 
determine  if  changing  the  values  will  result  in  a  different  outcome  of  the  assessment. 
There  are  two  possible  outcomes  of  the  assessment:  CSSQT  Required  or  No  CSSQT 
Required.  Out  of  the  ten  ships  assessed,  six  resulted  in  No  CSSQT  Required  with  the 
weighting  scheme  used  by  NSWC  PHD.  The  first  analysis  changed  the  weights  so  there 
was  a  single  magnitude  of  difference  between  the  categories  and  there  was  no  change  to 
the  outcome  of  the  assessment.  The  second  analysis  changed  the  value  of  the  weight  for 
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Necessary,  Like  to  Have  and  Non-Critical  systems  to  0.75,  0.50,  and  0.25,  respectfully, 
and  the  results  were  not  affected.  Each  analysis  maintained  the  weight  for  a  Critical 
system  as  1  and  the  values  for  the  other  weights  were  separated  by  an  equal  magnitude. 
The  analysis  was  performed  to  show  if  the  values  for  the  weight  influence  the  outcome  of 
the  assessment,  in  both  cases  the  outcome  was  not  affected. 

The  values  for  the  criticality  weights  do  not  influence  the  results  of  the 
assessment  and  the  methodology  presented  recommends  changing  the  values.  Changing 
the  weight  values  corrects  a  problem  with  how  the  overall  significance  is  calculated.  The 
current  process  multiplies  the  criticality  weight  by  the  modification  weight  to  detennine 
the  significance  a  change  has  on  an  individual  system.  When  a  non-critical  system  is 
being  modified,  it  does  not  factor  into  the  significance  because  a  Non-Critical  system  has 
a  weight  of  0.  To  maintain  an  equal  separation  of  weights,  a  Non-Critical  system  will 
have  a  weight  of  0.25,  a  Like  to  Have  system  a  weight  of  0.50,  and  a  Necessary  system  a 
weight  of  0.75.  The  process  for  including  systems  that  are  considered  Non-Critical  will 
be  discussed  in  the  section  describing  the  function  of  calculating  noon-critical 
significance. 


3.  Determine  Type  of  Modification 

The  type  of  modification  evaluates  if  a  systems  is  being  altered  during  the 
availability  and  the  nature  of  the  alteration.  There  are  five  categories  of  modifications,  the 
categories  and  the  descriptions  are  shown  in  Figure  13. 


Type 

Comments 

Example 

Replace  with  New 

This  is  a  significant  change  to  the  combat 
system.  Disrupts  the  DTE  data  path 

Replace  ACDS  with  SSDS,  SYS- 
2  with  CEC 

Add  Capability 

This  is  a  significant  change/addition  to  a 
mission  area 

Add  CIWS  to  a  platform, 
NSSMS  surface  mode 

Improve  Capability 

This  is  a  significant  change  to  a  systems 
capability 

ESSM,  IFF  mode  V 

Improve  Performance 

This  is  a  significant  change  to  a  systems 
performance 

NSSMS  solid  state 
transmitter,  48G 

Improve  Infrastructures 

This  is  a  significant  change  to  a  systems 
configuration 

SSDS  open  architecture 

Figure  12.  Modification  Types  and  Descriptions  (From  B.  Hazel  and 
S.  Matthews,  unpublished  Excel  spreadsheet) 
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The  main  issue  with  the  type  of  modification  categories  is  there  is  not  a  category 
for  a  system  not  being  changed.  This  leads  to  the  decision  tool  having  blank  cells  that 
gives  no  information  to  the  user.  A  blank  cell  could  indicate  that  the  system  has  not  been 
evaluated  or  the  type  of  change  is  unknown.  A  sixth  category  needs  to  be  added  to 
account  for  systems  not  being  changed  during  the  availability. 

a.  Apply  Modification  Weights 

The  purpose  of  applying  weights  to  the  modification  type  is  to  quantify  the 
impact  the  modification  has  on  the  system.  As  discussed  previously,  the  modification 
weight  and  the  criticality  weight  are  used  to  score  the  significance  of  the  changes  to  a 
warfare  area.  The  weighting  scheme  used  by  NSWC  PHD  give  equal  value  to  the  first 
three  types  of  modifications  indicating  the  impact  to  the  system  is  the  same  for  each 
modification.  The  last  two  categories  have  a  lesser  amount  of  weight  indicating  the 
modification  does  not  impact  the  system  as  much  as  the  other  modifications.  The  same 
scale  used  for  the  criticality  weights  is  used  for  the  modification  weights.  The  first  three 
modification  types  are  assigned  a  weight  of  1,  the  fourth  type  has  a  value  of  0.50  and  the 
last  category  is  given  a  weight  of  0.25.  Again,  there  is  an  unequal  separation  between  the 
types  of  modifications.  The  weighting  scheme  indicates  a  modification  that  replaces  an 
old  system  or  adds  or  improves  the  capability  of  a  system  will  impact  the  system  twice  as 
much  as  a  modification  that  improves  the  perfonnance  of  the  system  and  an  improvement 
to  performance  has  twice  the  impact  as  an  improvement  to  infrastructure.  Since  the 
weights  are  assigned  based  on  the  potential  impact  the  modification  has  on  a  system,  it  is 
possible  the  outcome  of  the  assessment  can  be  impacted  by  changing  the  values  of  the 
weights. 

Sensitivity  analysis  was  performed  to  determine  if  the  values  of  the 
weights  are  critical  to  the  outcome  of  the  assessment.  Two  analyses  were  conducted  to 
determine  if  the  recommendations  of  the  Decision  Tool  can  be  changed  if  the  values  for 
the  weights  are  altered.  The  first  analysis  increased  the  values  of  the  Improve 
Performance  and  Improve  Infrastructure  to  0.90  and  0.50,  respectfully.  The  second 
analysis  decreased  the  weights  to  0.30  and  0.10.  The  weights  of  the  first  three  types  of 
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modifications  were  not  changed  because  it  is  understood  the  impact  to  the  system  is  the 
greatest  when  one  of  these  modifications  occur.  The  outcome  of  the  assessments  of  ten 
ships  did  not  change  in  either  of  the  two  analyses.  Further  analysis  would  be  required  if 
the  outcomes  did  change  to  determine  the  sensitivity  of  the  model.  This  simple  sensitivity 
analysis  demonstrates  the  model  is  not  sensitive  to  changing  the  weights  and  therefore, 
the  weights  will  not  be  changed. 

4.  Calculate  Warfare  Area  Significance 

The  purpose  of  this  function  is  to  determine  the  magnitude  of  impact  the  changes 
to  the  systems  has  on  a  warfare  area.  This  is  computed  by  multiplying  the  criticality 
weight  by  the  modernization  weight.  If  a  system  is  not  affected  during  the  availability, 
the  score  will  be  zero  because  the  weight  of  no  change  is  zero.  The  highest  score  an 
individual  system  can  receive  is  one.  The  sum  of  all  the  individual  system  scores  will 
yield  the  impact  to  the  warfare  area.  This  value  will  range  from  0  to  10  (in  most  cases, 
there  is  the  possibility  that  the  value  could  be  greater  than  10,  but  it  is  not  likely)  where 
10  indicates  a  significant  amount  of  modification  is  being  made  to  the  combat/weapon 
systems.  The  review  found  the  current  process  does  this  calculation,  but  does  another 
calculation  that  is  not  necessary. 

NSWC  PHD  detennines  the  significance  of  changes  made  to  each  warfare  area  by 
taking  the  sum  of  individual  impact  scores  and  dividing  it  by  the  sum  of  the  criticality 
weights.  The  ratio  of  warfare  area  impact  to  warfare  area  criticality  yields  a  value  that  is 
converted  to  a  percentage.  The  percentage  is  what  is  used  to  indicate  the  amount  of 
change.  The  flaws  with  this  method  are,  1)  it  does  not  capture  non-critical  systems  that 
are  being  altered  during  the  availability,  2)  a  ratio  is  intended  to  compare  two  similar 
values  and  in  this  model  the  two  values  are  not  similar  and  3)  the  result  is  in  a  form  of  a 
percentage  but  it  is  unclear  what  it  is  a  percentage  represents.  The  solution  to  the  first 
flaw  is  to  calculate  the  significance  to  non-critical  systems  which  will  be  discussed  in  the 
next  section.  The  last  flaws  can  be  resolved  by  not  doing  the  calculation  and  only  using 
the  sum  of  the  warfare  area  significance  scores. 
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5.  Calculate  Non-Critical  System  Significance 

The  purpose  of  this  function  is  to  account  for  systems  that  are  not  critical  to  a 
major  warfare  area.  All  combat/weapon  systems  are  important  to  the  overall  mission  of 
the  ship  but  some  will  not  directly  support  primary  warfare  areas.  Changes  to  these 
systems  must  be  accounted  for  to  detennine  if  the  modernization  availability  will  have 
significant  changes.  The  reason  for  accounting  for  non-critical  systems  is  because  any 
change  will  impact  the  crew’s  ability  to  operate  the  equipment.  For  example,  a  hardware 
change  to  the  air  traffic  control  system  may  not  impact  air  defense  or  surface  warfare,  but 
it  does  impact  the  ship’s  ability  to  recover  aircraft.  These  non-critical  systems  must  be 
included  in  the  overall  assessment  of  significance.  The  non-critical  system  significance  is 
calculated  in  the  same  way  the  warfare  area  significance  is  calculated;  how  it  is 
implemented  will  be  discussed  in  Chapter  IV. 

B.  STAGE  2 

The  risk  assessment  tool  process  addresses  risk  to  a  ship’s  readiness  after  an 
overhaul/modemization  availability.  The  current  process  used  by  NSWC  PHD  is  still 
under  development  and  has  not  been  implemented  to  assess  the  risk  for  any  ship 
conducting  a  modernization  availability.  For  this  reason,  the  best  approach  is  to 
determine  a  way  to  assess  the  risk  associated  with  a  major  overhaul.  After  a  practical 
method  is  developed  to  determine  the  risk,  the  next  step  is  to  review  the  proposed  process 
discussed  in  Chapter  II. 

1.  Risk  Assessment  Methodology 

The  difficulty  with  risk  assessment  is  trying  to  predict  the  future  based  on  current 
knowledge.  The  first  step  for  assessing  risks  requires  understanding  all  the  possible 
outcomes,  determining  what  can  cause  those  outcomes,  and  the  probability  of  each  cause 
occurring.  The  next  step  is  to  apply  a  severity  to  each  outcome;  the  greater  the  severity, 
the  greater  the  risk.  Risk  is  defined  as  a  measure  of  the  probability  and  severity  of 
adverse  effects  (Haimes  2009,  4).  There  are  many  methods  that  can  be  used  to  assess  risk, 
but  all  methods  essentially  start  with  the  following  three  questions: 

•  What  can  go  wrong? 
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•  What  is  the  likelihood  that  it  would  go  wrong? 

•  What  are  the  consequences?  (Haimes  2009,  24) 

The  methodology  for  this  research  will  begin  with  these  three  questions.  To  begin 
the  assessment  it  is  important  to  understand  the  goal  of  CSSQT.  Figure  14  shows  that  the 
goal  of  CSSQT  is  to  ensure  a  safe  and  effective  combat  system. 


Goal:  Safe  and  Effective  Combat  System 


Accomplished  by:  PMS/MRC  and  Live/ Simulated  Exercises 


Maintenance 

Readiness 


Operational 

Readiness 


Safety 


• 

PMS/MRC 

•  Live/Simulated 

•  Ordnance  Handling 

• 

System  Documentation 

Operational 

•  Ordnance  Movement 

Adequacy 

Exercises 

•  Loading  Procedures 
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Logistics  Support 

•  PEO  Requirements 

Adequacy 

•  TYCOM 
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Maintenance  Adequacy 

Requirements 
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Operational  Adequacy 

Figure  13.  CSSQT  Goals,  Objectives,  and  Requirements  (From  NAVSEA  2006) 


The  three  areas  that  ensure  a  safe  and  effective  combat  system  are  maintenance 
readiness,  operational  readiness  and  safety.  Figure  13  provides  categories  for  answers  to 
the  first  question  of  “What  can  go  wrong?”  They  include:  the  ship  may  not  be  able  to 
accomplish  maintenance  actions,  the  ship  may  not  be  operationally  ready,  and  the  crew 
may  not  be  able  to  safely  handle  ammunition.  The  third  question  focuses  on  the 
consequences  from  the  answers  to  what  can  go  wrong.  If  a  ship  is  not  operationally 
ready,  then  the  training  cycle  may  take  longer  or  require  more  man  hours  and  could  lead 
to  schedule  changes  or  delayed  deployments.  If  the  crew  is  unable  to  maintain  the  new 
equipment,  then  the  consequences  could  be  increased  technical  assistance,  and  longer 
equipment  downtime.  This  risk  assessment  will  not  focus  on  the  consequences  but  only, 
address  the  likelihood  of  what  can  go  wrong. 

Answering  the  second  question  about  the  likelihood  of  a  risk  occurring  is  what 
this  risk  assessment  is  going  to  accomplish.  CSSQT  ensures  the  ship  has  a  safe  and 
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effective  combat  system  after  significant  overhaul/modemization,  but  what  is  the 
likelihood  of  a  ship  not  being  safe  and  effective  if  a  CSSQT  is  not  conducted?  The 
significance  assessment  may  result  in  an  overhaul/modernization  being  classified  as 
significant  but,  the  assessment  is  based  on  the  type  of  changes  and  how  the  changes 
impact  the  warfare  areas.  The  risk  assessment  addresses  how  the  entire  availability,  not 
only  the  modifications,  affect  the  three  areas  reviewed  by  CSSQT.  If  the  maintenance 
availability  does  not  impact  the  maintenance  readiness,  operational  readiness,  or  safety  of 
the  ship,  than  it  may  not  be  necessary  to  conduct  a  CSSQT  even  if  the  availability  is 
deemed  significant.  Cause-Effect  diagraming  will  be  used  to  detennine  how  the 
availability,  as  a  whole,  impact  the  three  focuses  of  CSSQT. 

During  the  availability,  what  is  being  added  or  changed,  how  long  is  the 
availability,  what  is  the  time  lapse  since  training  cycle,  or  whether  there  is  crew  turnover 
can  have  great  impact  on  how  well  the  ship’s  force  can  maintain  and  operate  its  new 
combat  system.  A  cause  and  effect  diagram  can  be  used  to  show  which  areas  are  at  risk. 
Figure  15  depicts  how  a  safe  and  effective  combat  system  is  achieved  by  conducting  a 
CSSQT. 


Ensure  Maintenance  Readiness 


CSSQT  Cause  and  Effect  Diagrarr 


-  Maintenance  Adequacy 


Ensure  Operational  Readiness 


Software  Change 


Hardware  Change 


Percentage  of  Systems  Changed 

/Tech  Manuals 

Documentation  Live/Simulated  Operational  ^ 


Length  of  Availability 


Years  Since  Last  CSSQT 


*  ►[  Safe  and  Effective  Combat  Systerrj 


-  Number  of  Weapon  Systems  Changed 


Ensure  Safety 


Figure  14.  Cause-Effect  Diagram 
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Cause  and  Effect  diagrams  are  usually  used  to  find  the  root  cause  of  a  problem, 
but  in  this  case  the  goal  is  to  find  what  influences  a  positive  outcome.  The  diagram  can 
also  be  used  to  determine  the  risks  to  the  causes.  Each  cause  can  be  broken  down  into 
secondary  causes  and  these  causes  can  be  used  to  detennine  what  drives  each  primary 
cause.  For  example,  CSSQT  ensures  operational  readiness  by  performing  live  and/or 
simulated  operational  exercises.  If  operational  exercises  are  not  performed  by  the  crew 
then  operational  readiness  cannot  be  ensured;  therefore,  the  ship  has  a  risk  of  not  having  a 
safe  and  effective  combat  system.  The  secondary  cause  can  be  broken  down  further  (if 
required)  to  address  potential  risk  factors  that  can  be  used  to  assess  the  risk  to  each 
primary  cause.  Each  arrow  on  the  diagram  represents  a  risk  factor.  Each  risk  factor  needs 
to  be  investigated  to  detennine  a  way  to  quantify  the  risk. 

The  three  area  of  focus  are  maintenance  readiness,  operational  readiness,  and 
safety;  these  are  the  risk  areas  for  overall  ship  readiness.  Measuring  the  risk  to  the  ship’s 
readiness  will  be  a  function  of  the  risk  to  the  three  risk  areas.  The  goal  is  to  quantify  the 
risk  in  a  way  the  represents  the  likelihood  of  risk;  that  is,  the  higher  the  value  the  more 
likely  the  ship  will  not  be  operationally  ready.  The  next  sections  will  break  down  each 
risk  area  and  discuss  measuring  the  risk. 

a.  Risk  to  Maintenance  Readiness 

Maintenance  readiness  is  the  ship’s  ability  to  maintain  the  combat/weapon 
systems  onboard.  Prior  to  entering  a  major  availability,  the  crew  has  the  right  people 
trained  to  maintain  the  equipment  and  will  have  accumulated  months  of  experience 
maintaining  it.  After  the  availability,  the  maintenance  readiness  may  be  decreased 
because  a  system  that  has  been  upgraded,  replaced  or  installed  may  affect  the  hardware, 
software,  planned  maintenance  system,  technical  manuals,  supply  parts,  certifications, 
Operating  Sequencing  System  (OSS),  detect-to-engage  path,  training,  and  test 
equipment.  These  are  the  risk  factors  taken  from  the  cause  and  effect  diagram.  Not  all  of 
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these  factors  will  be  affected,  so  the  assessment  of  risk  to  maintenance  readiness  will 
determine  what  is  being  changed  and  quantify  the  changes. 

The  main  source  of  data  for  impact  to  maintenance  readiness  is  a  SCD.  As 
discussed  previously,  SCDs  contain  information  about  administrative  functions  and  the 
impact  on  maintenance  readiness.  A  SCD  consists  of  26  sections  and  the  following 
sections  are  used  to  determine  the  risk  to  maintenance  readiness: 

•  Section  4a  and  4b:  Scope  and  Category 

•  Section  1 1 :  Executive  Summary 

•  Section  12:  Description  of  Change 

•  Section  13:  Impact  if  not  Accomplished 

•  Section  14:  Requirements  and  Justification  of  Change 

•  Section  15:  Distributive  Systems/Other  Impacts 

•  Section  16:  Human  Systems  Integration  Impacts 

•  Section  17:  Integrated  Logistics  Support  (ILS)  Impact 

•  Section  23:  Certifications/Qualifications 

The  scope  and  category  will  indicate  if  it  is  a  hardware  change,  software 
change  or  both.  Sections  11,  12,  13,  and  14  can  all  be  used  to  understand  the  nature  of  the 
change,  such  as  if  a  new  system  is  replacing  an  old  system.  Section  15  describes  how  the 
alteration  impacts  other  systems  on  the  ship.  Section  16  will  indicate  how  training  is 
impacted.  Section  17  lists  all  ILS  systems  that  are  impacted  and  Section  23  lists  the 
certifications  that  will  be  required  after  the  installation.  SCDs,  in  most  cases,  do  not 
describe  the  impact  a  change  will  have  on  a  specific  maintenance  area. 

Due  to  the  size  of  an  SCD  only  one  section  is  presented  here  to 
demonstrate  how  the  information  is  presented.  Ligure  16  is  an  example  of  what  is  found 
in  section  17  of  a  SCD.  The  only  information  that  can  be  inferred  from  this  section  is  the 
change  will  affect  technical  manuals  and  provisioning. 
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Figure  15. 

SCD  Section  17:  Integrated  Logistics  Support  (ILS)  Impact 

It  is  not  known  if  technical  manuals  will  be  added,  removed  or  updated  or  how  the 
supply  systems  (provisioning)  will  be  impacted.  If  this  information  was  available  it  may 
be  possible  to  use  an  assessment  approach  that  uses  the  number  of  changes  being  made  to 
each  area.  For  example,  if  the  system  before  the  change  required  45  spare  parts  onboard 
and  after  the  change  the  system  will  require  60  spare  parts  than  the  33%  increase  in  the 
number  of  parts  could  be  used  to  indicate  a  certain  likelihood  that  maintenance  readiness 
will  be  impacted.  The  reason  maintenance  readiness  could  be  impacted  is  because  of  the 
possibility  the  supply  system  does  not  get  updated  with  the  correct  parts.  If  the  ship  is  not 
stocked  with  the  correct  parts  than  a  delay  in  repair  could  occur  causing  longer  downtime 
of  the  system.  CSSQT  verifies  that  the  ship  has  the  correct  parts  onboard  for  each  system 
therefore  reducing  the  likelihood  of  this  situation  occurring.  The  assessment,  if  this 
information  was  present,  could  set  thresholds  for  the  amount  of  change  being  made  to  an 
area  and  make  a  better  determination  of  risk  if  the  threshold  is  exceeded. 

Information  on  how  each  area  is  being  affected  is  not  available  therefore,  a  binary 
system  is  the  best  approach  to  evaluate  the  changes  being  made  to  the  system;  if  a  box  is 
checked  then  the  category  receives  a  one,  if  not  checked  it  receives  a  0.  This  method  can 
be  used  on  each  system  being  impacted  by  the  availability.  There  are  ten  categories  that 
relate  to  maintenance  readiness,  so  if  a  specific  system  scores  a  10  then  all  areas  are 
being  affected  by  the  change.  The  average  score  of  the  all  the  systems  will  yield  a  value 
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that  is  indicative  of  how  much  the  availability  is  impacting  the  maintenance  readiness. 
The  higher  the  score,  the  more  likely  there  will  be  a  risk  to  maintenance  readiness  post¬ 
availability. 


b.  Risk  to  Operational  Readiness 

Operational  readiness  is  the  adequacy  at  which  the  ship’s  force  can 
operate  its  combat  system  and  the  ability  of  the  equipment  to  operate  as  designed.  The 
secondary  indicator  of  operational  readiness  is  perfonnance  of  live  and/or  simulated 
operational  exercises.  When  a  ship  conducts  a  CSSQT  it  will  perfonn  multiple  simulated 
warfare  specific  scenarios  that  stress  the  combat/weapon  systems  and  the  ship’s  crew. 
The  final  exercises  will  usually  involve  live  firing  of  missiles  and  guns  against  unmanned 
drones  and  other  targets.  Most  ships  will  only  get  to  conduct  live  firing  during  CSSQT, 
but  all  ships  conduct  simulated  exercises  during  the  training  cycle  and  during 
deployments.  Detennining  the  risk  to  operational  readiness  involves  these  two  aspects  of 
a  ship’s  life:  Time  lapse  since  last  CSSQT  and  length  of  availability.  The  latter  is  under 
the  assumption  that  a  majority  of  ships  return  from  deployment  and  enter  the 
maintenance  phase.  A  third  aspect  that  can  be  used  to  detennine  operational  readiness 
risk  is  the  percentage  of  systems  modified  during  an  availability.  The  more  systems  that 
are  changed,  in  most  cases,  the  less  familiar  the  crew  will  be  with  operating  them.  The 
ability  to  use  the  systems  as  designed  in  a  stressing  situation  decreases  because  of  the 
novelty  of  the  revised  system.  The  likelihood  of  risk  to  operational  readiness  will  be  a 
function  of  time  lapse,  length  of  availability  and  percentage  of  systems  changed.  The 
likelihood  will  range  from  0  to  10  where  10  would  indicate  the  greatest  likelihood. 

c.  Risk  to  Safety 

The  area  of  safety  CSSQT  addresses  is  ordnance  safety.  Ordnance 
handling,  movement,  and  loading  is  a  high  risk  evolution  and  if  a  ship  has  not  had 
training  on  the  proper  procedures  for  an  extended  period  of  time,  the  risk  is  increased. 
What  can  go  wrong  is  the  mishandling  of  ordnance  or  improper  loading  procedures 
which  can  lead  to  mishaps  that  can  cause  injury  or  even  death.  The  likelihood  of  the  ship 
not  being  safe  is  a  function  of  the  weapons  systems  being  altered  during  the  availability. 
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If  an  availability  will  affect  all  weapon  systems  onboard  than  the  likelihood  of  risk  is  the 
greatest.  In  this  case  the  ship  will  receive  a  score  of  10.  If  no  weapon  systems  are  being 
upgraded  or  added  then  the  risk  is  the  lowest  yielding  a  score  of  0.  When  there  is  one  or 
more  weapon  system  is  being  modified,  but  not  all,  then  the  score  will  be  a  5. 

2.  Review  of  Current  Risk  Assessment  Process 

The  risk  assessment  methodology  shows  the  functions  required  for  the  assessment 
are  detennining  the  risk  areas,  the  risk  factors  for  each  area,  and  a  way  to  quantify  the 
risk  factors.  Another  function,  that  was  not  discussed,  is  the  ability  to  repeat  the 
assessment  in  a  way  that  will  ensure  consistent  results.  This  means  that  the  data  used  to 
conduct  the  assessment  must  lack  any  sort  of  subjective  assessment.  NSWC  PHD  has 
developed  a  process  to  assess  risk  and  it  was  described  in  Chapter  II.  This  process 
identified  four  risk  areas  and  18  risk  factors.  It  also  quantified  the  risk  factors  so  they 
could  be  used  to  formulate  a  value  for  likelihood.  The  issue  with  the  proposed  process  is 
that  it  relies  on  factors  that  are  subjective. 

There  are  five  risk  factors  in  the  proposed  process  that  solicit  an  evaluation  of  a 
SME.  The  five  factors  are: 

•  Ship’s  force  ability  to  perfonn  the  mission 

•  Known  system  of  incidents  of  risks 

•  Known  safety  incidents  of  risks 

•  Project  Officer/Project  Engineer  evaluation  of  risk 

•  Baseline  Manager/Equivalent  evaluation  of  risk 

The  evaluation  by  SMEs  can  differ  greatly  from  ship  to  ship  and  it  is  important  all 
subjective  assessments  are  removed  when  conducting  a  quantitative  assessment.  It  also 
introduced  a  level  of  complexity  because  the  execution  of  the  assessment  relies  on  the 
input  of  outside  entities.  There  are  other  risk  factors  that  rely  on  the  input  from  other 
agencies  to  complete  the  assessment  and  those  are: 

•  Training  program  planned  or  developed/delivered(AIT) 

•  Training  program  planned  or  developed/delivered(LBTS) 

•  Planned  ATG  Training 
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•  Manpower  requirements  adequately  reflected 
The  problem  with  using  the  inputs  from  outside  sources  is  potential  delays  in  gathering 
information.  In  any  given  fiscal  year  there  will  a  large  number  of  ships  conducting  an 
availability  and  it  is  the  advantage  to  the  assessors  to  rely  on  one  or  two  sources  for  data. 
The  current  process  could  take  an  assessor  weeks  to  complete  while  the  methodology 
described  in  the  previous  section  is  able  to  be  completed  in  days. 

The  other  issue  with  the  risk  assessment  process  proposed  by  NSWC  PHD  is  that 
it  does  not  follow  the  goals  of  the  NAVSEAINST  9093. 1C.  The  instruction  specifically 
outlines  what  CSSQT  accomplishes  and  the  risk  assessment  should  coincide  with  the 
objectives  of  the  instruction.  There  are  only  three  areas  that  CSSQT  addresses  but  the 
proposed  process  describes  four  areas.  Each  area  in  the  instruction  had  specific  factors 
that  are  reviewed  by  the  CSSQT  but  the  proposed  process  has  multiple  factors  that  apply 
to  more  than  one  of  the  four  areas.  The  risk  assessment  should  be  precise  and  lack  any 
ambiguity;  however,  the  proposed  risk  assessment  process  is  full  of  ambiguity  and  does 
not  provide  a  precise  measure  of  the  likelihood  that  the  ship’s  readiness  is  at  risk. 

C.  STAGE  3 

The  purpose  of  this  stage  is  to  develop  a  methodology  for  presenting  the  results 
from  the  significance  assessment  methodology  and  risk  assessment  methodology.  The 
current  process  used  by  NSWC  PHD  uses  a  separate  spreadsheet  to  assess  the 
significance  and  another  one  for  the  risk  assessment.  The  only  interface  between  the  two 
processes  is  that  the  result  of  the  significance  assessment  is  used  in  the  risk  assessment  as 
a  risk  factor.  NSWC  PHD  has  proposed  that  the  results  of  the  assessment  will  be  reported 
on  a  single  page  that  is  divided  into  four  quadrants  (commonly  referred  to  as  a  quad- 
slide)  where  one  quadrant  gives  information  about  the  ship,  another  one  for 
administrative  data,  and  the  last  two  show  the  results  of  the  assessments.  The  issue  with 
this  approach  is  that  there  is  no  coupling  of  results  of  the  assessments. 

The  end  result  of  the  two  assessments  should  result  in  an  understanding  of  the 
overall  impact  to  the  ship.  Overall  impact  is  a  function  of  the  how  significant  the  changes 
are  and  the  likelihood  that  the  changes  will  affect  the  ship’s  readiness.  The  presented 
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methodologies  put  significance  and  likelihood  on  the  same  sized  scale  so  it  makes  it 
possible  plot  the  two  areas  in  a  matrix.  The  reason  for  plotting  the  two  areas  is  to  give  a 
visual  representation  of  the  results.  When  each  ship  is  plotted  on  the  same  matrix, 
decision-makers  will  have  the  ability  to  determine  which  ship  will  be  impacted  the 
greatest  and  also  know  which  ships  can  proceed  without  a  CSSQT.  The  approach  for 
presenting  the  results  will  be  to  use  a  summary  spreadsheet  that  captures  each  ship 
conducting  an  availability  in  a  given  fiscal  year,  infonnation  about  each  ship’s 
availability  and  history,  the  results  of  the  significance  assessment,  the  results  of  the 
likelihood  assessment  and  a  matrix  that  plots  each  ship. 

This  chapter  developed  methodologies  for  determining  if  the  changes  being  made 
to  a  ship  during  a  modernization  are  significant,  if  the  changes  being  made  impact  the 
readiness  of  the  ship,  and  a  way  to  present  the  results.  It  also  reviewed  the  current  tools 
being  used  by  NSWC  PHD  to  determine  shortfalls  and  issues.  The  review  found  that 
there  are  issues  with  the  current  tools  that  need  to  be  addressed.  The  next  chapter  will  use 
the  methodologies  to  develop  a  tool  that  will  determine  significance  and  likelihood  and 
present  the  results. 
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IV.  RESULTS  AND  ANALYSIS 


The  previous  chapter  discussed  issues  with  the  current  process  for  detennining  the 
impact  to  a  ship’s  readiness  after  a  major  availability  and  also  developed  methodologies 
to  implement  a  better  process.  The  results  of  implementing  the  methodologies  will  be 
described  in  this  chapter.  The  first  step  designs  a  spreadsheet  that  presents  the 
information  about  each  ship  being  assessed,  the  results  of  the  significance  assessment, 
and  the  results  of  the  risk  assessment.  The  second  step  is  to  conduct  the  significance 
assessment  and  risk  to  maintenance  readiness  assessment.  The  final  step  plots  the 
significance  and  risk  likelihood  for  each  ship  on  a  matrix.  The  end  product  will  be  a 
single  spreadsheet  that  presents  the  results  of  the  significance  assessment,  risk  assessment 
and  matrix. 

The  analysis  will  use  data  about  ships  that  are  scheduled  to  conduct  a  major 
availability  in  fiscal  year  2015.  The  ships  used  for  the  assessment  are: 

•  USS  Nimitz  (CVN  68) 

•  USS  George  HW  Bush  (CVN  77) 

•  USS  Harry  S  Truman  (CVN  75) 

•  USS  Ronald  Reagan  (CVN  76) 

•  USS  Germantown  (LSD  42) 

•  USS  Gunston  Hall  (LSD  44) 

•  USS  Ashland  (LSD  48) 

•  USS  Carter  Hall  (LSD  50) 

•  USS  Pearl  Harbor  (LSD  52) 

•  USS  Makin  Island  (LHD  8) 

All  infonnation  needed  to  conduct  the  assessment  is  available  in  three  locations: 
WISE  Database,  Navy  Data  Environment  (NDE),  and  NSWC  PHD  ship  information 
database.  The  WISE  database  contains  infonnation  about  the  availability  and  the  systems 
onboard  the  ship.  SCDs  are  located  in  the  NDE  and  the  ship  information  database  has  the 
results  of  the  previously  conducted  CSSQT. 
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A.  CSSQT  DETERMINATION  DASHBOARD 

The  CSSQT  Determination  Dashboard  is  a  central  location  that  presents  the 
information  gathered  during  the  assessment.  The  dashboard  uses  information  about  the 
availability,  the  results  of  the  assessments  and  does  calculations  to  determine  the 
significance  of  changes  and  risk  likelihood.  Figure  17  shows  what  a  completed  dashboard 
looks  like. 


FY15 


CSSQT  Determination  Dashboard 


Availability  Data 

Risk  Assessment 

Ship 

Avail 

Avail  Start 

Avail  End 

AD 

Impact 

Score 

suw 

Impact 

Score 

Non-Crit 

Weight 

Significanc 

e  Score 

Avail 

Length 

(Years) 

Total 

Systems 

Total 

Systems 

Changed 

% 

Systems 

Changed 

Last 

CSSQT 

Delta 

(Years) 

Operational 

Readiness 

Score 

Maintenanc 

e  Readiness 

Score 

Safety 

Score 

Risk 

Likelihood 

CVN  68 

PA3 

May -14 

Feb-15 

5.13 

4.13 

0.31 

4.94 

0.75 

29 

13 

45% 

2007 

8 

2.77 

6.77 

10 

6.51 

P1A1 

Apr- 15 

Oct-15 

3.88 

C.69 

5.44 

0.50 

31 

13 

42% 

6 

1.94 

7.C8 

10 

6.34 

SRA 

Jan-15 

Apr-15 

1.0C 

1.0C 

C.OC 

1.00 

a25 

18 

2 

11% 

1986 

29 

1.11 

5.5 

1 

2.54 

■339 

5.0C 

4.75 

069 

5.56 

a33 

20 

14 

70% 

1989 

26 

7.00 

6.57 

10 

7.86 

LSD  *8 

SRA 

NOv-14 

2.5C 

3. 13 

0.00 

2.81 

a25 

19 

7 

37% 

1992 

22 

3.68 

5.71 

3.47 

LSD  50 

■539 

6.25 

6.0C 

C.75 

6.88 

1.00 

20 

15 

75% 

1995 

20 

7.50 

6. 80 

10 

8.10 

LtiAM 

■239 

6.25 

5. 13 

M-IM 

6.31 

1.00 

20 

14 

70% 

1998 

17 

7.00 

6. 80 

10 

7.93 

CVN  75 

PlA2 

5.5C 

3.0C 

1.25 

5.50 

0.50 

31 

16 

52% 

8 

3.16 

7.CC 

10 

6.72 

i-tvfri-a 

PtA2 

E!39 

5.0C 

4.0C 

2.00 

6.50 

aso 

32 

16 

50% 

K»V1 

7 

2.69 

6.75 

10 

6.48 

U,Y_V>« 

3.75 

3.  SC 

0.13 

29 

8 

28% 

1.28 

7.5 

10 

6.26 

Figure  16.  Completed  CSSQT  Determination  Assessment 


Figure  16  is  used  to  show  the  design  of  the  dashboard;  but,  each  section  will 
discussed  in  depth  for  better  understanding. 

1.  Availability  Data 

Figure  18  shows  the  first  section  in  the  Determination  Dashboard.  The 
information  in  this  section  is  found  in  the  WISE  database.  It  lists  the  ship’s  hull  number, 
the  type  availability,  and  the  start  and  end  date  of  the  availability.  Each  ship  is 
hyperlinked  to  the  ship  specific  worksheet  that  contains  the  significance  assessment  and 
risk  to  maintenance  assessment. 


Availability  Data 

Ship 

Avail 

Avail  Start 

Avail  End 

Figure  17.  Availability  Data  Section  of  Dashboard 
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2.  Significance  Assessment 

Figure  19  shows  the  areas  that  make  up  the  significance  assessment  section  of  the 
dashboard.  AD  Impact  Score,  SUW  Impact  Score  and  the  Non-Critical  Weight  are 
extracted  from  the  significance  assessment  worksheet  for  each  ship.  The  significance 
score  is  the  average  of  the  AD  Impact  Score  and  SUW  Impact  Score  plus  the  Non- 
Critical  Weight. 


Significance  Assessment 

AD 

Impact 

Score 

SUW 

Impact 

Score 

Non-Crit 

Weight 

Significance 

Score 

Figure  18.  Significance  Assessment  Section  of  Dashboard 

3.  Risk  Assessment 

Figure  20  shows  the  areas  that  make  up  the  risk  assessment  section  of  the 
dashboard 


Risk  Assessment 


Avail 

Total 

% 

Operational 

Maintenance 

Length 

Total 

Systems 

Systems 

Last 

Delta  Readiness 

Readiness 

Safety 

Risk 

(Months) 

Systems 

Changed 

Changed 

CSSQT 

(Years)  Score 

Score 

Score 

Likelihood 

Figure  19.  Risk  Assessment  Section  of  Dashboard 


Availability  length  is  the  number  of  months  of  the  availability.  Total  Systems  is  a 
count  of  the  number  of  combat/weapon  systems  onboard  the  ship.  Total  Systems 
Changed  is  number  of  systems  that  are  being  changed  or  added  during  the  availability. 
Both  inputs  are  calculated  from  the  significance  assessment  and  the  percentage  of 
systems  changed  is  the  total  systems  changed  divided  by  the  total  systems.  Last  CSSQT 
is  the  previous  year  the  ship  conducted  a  CSSQT.  The  number  of  years  that  have  elapsed 
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since  the  last  CSSQT  and  the  end  of  the  availability  is  the  Delta.  The  next  four  areas  of 
the  risk  assessment  section  are  calculated  from  the  previous  sections  and  also  use  input 
from  a  separate  worksheet. 

Operational  Readiness  score  is  a  function  of  Availability  Length,  Percentage  of 
Systems  Changed,  and  Delta.  The  function  is  designed  to  give  more  weight  to  the  time 
elapse  since  the  last  CSSQT.  This  is  accomplished  by  weighting  the  sum  of  the  Delta  and 
length  of  availability.  If  the  Delta  is  greater  than  13  years  the  Operational  Readiness 
score  will  be  dependent  on  the  percentage  of  systems  changed.  So,  if  a  large  percentage 
of  systems  are  being  affected  and  the  delta  between  CSSQTs  is  greater  than  13  years,  the 
likelihood  of  risk  to  operational  readiness  is  greater.  When  the  elapse  between  CSSQTs  is 
less  than  13  years  the  operational  readiness  risk  likelihood  is  less.  The  formula  used  to 
determine  the  Operational  Readiness  score  in  an  Excel  spreadsheet  is 
MIN(10,LengthOfAvailabilty*0.25+Delta*0.75)*%SystemsChanged.  The  formula  is 
designed  to  ensure  the  score  of  Operational  Readiness  is  less  than  or  equal  to  10.  The 
next  areas  are  the  risk  to  Maintenance  Readiness  and  Safety. 

The  risk  to  Maintenance  Readiness  Score  comes  from  an  assessment  that  is 
performed  on  the  Significance  Assessment  worksheet.  The  details  of  the  assessment 
procedure  will  be  discussed  in  the  next  section.  The  Safety  Score  will  be  a  0,  5,  or  10 
based  on  the  number  of  weapon  systems  being  altered  during  the  availability.  This  score 
is  detennined  by  the  user  and  can  be  found  by  looking  at  the  Significance  Assessment  to 
find  which  weapon  systems  are  being  modified.  The  Safety  Score  can  also  be  a  10  if  the 
Significance  Assessment  shows  that  a  new  weapon  system  is  being  installed  on  the  ship 
that  will  require  new  ammunition  handling  and  loading  procedures.  The  final  value  in  the 
Risk  Assessment  section  is  the  overall  Risk  Likelihood.  It  is  the  average  of  the 
Operational  Readiness,  Maintenance  Readiness  and  Safety  scores. 
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B.  SIGNIFICANCE  ASSESSMENT  AND  RISK  TO  MAINTENANCE 

READINESS  ASSESSMENT 

Four  values  in  the  Determination  Dashboard  are  determined  on  a  separate 
worksheet  within  the  spreadsheet  that  are  specific  to  each  ship  being  assessed.  The 
significance  assessment  will  calculate  how  significant  the  changes  are  to  each  warfare 
area  and  non-critical  systems.  The  maintenance  readiness  assessment  will  calculate  the 
likelihood  of  risk  to  maintenance. 

1.  Significance  Assessment 

NSWC  PHD  called  this  assessment  the  CSSQT  Decision  Tool  Process.  The 
review  found  areas  of  the  process  that  needed  improvement  and  the  name  is  the  first  issue 
that  needs  to  be  addressed.  The  process  is  not  making  a  decision;  rather,  it  is  assessing  if 
the  changes  being  made  to  the  combat/weapon  systems  are  significant.  It  is  a  minor 
change,  but  gives  more  clarity  to  the  purpose  of  the  process.  The  second  issue  found  was 
the  accuracy  of  the  systems  installed  onboard  each  ship.  There  were  numerous  ships  that 
had  the  wrong  systems  listed  and/or  systems  were  missing.  This  is  easily  addressed  by 
double  checking  that  systems  input  on  the  worksheet  are  the  same  systems  listed  on  the 
WSID.  The  third  issue  was  the  mission  area  criticality  and  modernization  type  categories 
and  subsequent  weights  which  will  be  discussed  next. 

The  review  of  the  process  found  that  systems  that  were  not  essential  to  a  specific 
warfare  area  were  not  assessed  even  if  the  systems  were  being  altered.  All 
combat/weapon  systems  having  any  sort  of  modification  need  to  be  included  in  the 
assessment  regardless  of  the  importance  to  a  warfare  area.  This  issue  is  addressed  by 
giving  weight  to  non-critical  systems.  The  modernization  type  categories  did  not  have  a 
category  for  systems  not  being  changed  which  led  to  a  spreadsheet  that  contained  a 
number  of  blank  cells  and  cells  that  contained  FALSE  as  an  entry.  When  a  spreadsheet 
analysis  has  blank  cells  and  FALSE  cells,  it  looks  incomplete.  This  is  addressed  by 
adding  a  sixth  category  to  the  modernization  types.  The  new  categories  and  weights  used 
in  the  Significance  Assessment  are  shown  in  Figure  2 1 . 
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Modernization  Type 

Weight 

Replace  with  New 

l 

Add  Capability 

l 

Improve  Capability 

l 

Improve  Performance 

0.5 

Improve  Infrastructures 

0.25 

No  Change 

0 

Criticality 

Weight 

Critical 

l 

Necessary 

0.75 

Like  to  have 

0.5 

Non-Critical 

0.25 

Figure  20.  Updated  Categories  and  Weights 


The  final  issue  the  review  found  with  the  process  to  determine  if  the  changes  are 
significant  is  the  calculation  of  the  significance  value.  As  discussed  in  Chapter  III,  each 
system  received  a  score  by  multiplying  the  mission  criticality  weight  by  the 
modernization  type  weight.  The  sum  of  all  the  systems  scores  yielded  a  total  score  for  the 
warfare  area.  The  process  then  would  sum  the  warfare  area  criticality  weights  and  divide 
the  total  score  for  the  warfare  area  by  this  sum.  This  ratio  was  the  value  used  to  show 
how  the  changes  impact  each  warfare  area.  The  average  of  the  warfare  area  scores  would 
be  the  overall  impact  to  the  ship. 

The  methodology  in  Chapter  III  determines  a  better  way  to  calculate  if  the 
changes  are  significant  is  to  use  only  the  sum  of  the  system  scores  since  this  value  takes 
into  account  both  criticality  and  the  type  of  modernization.  Another  problem  with  the 
calculation  was  it  did  not  account  for  non-critical  systems.  The  score  for  non-critical 
systems  is  calculated  in  the  same  manner  as  the  score  for  a  warfare  area.  The  score  for 
each  warfare  area  and  non-critical  systems  is  used  in  the  Detennination  Dashboard  to 
calculate  the  overall  significance  of  changes. 
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The  assessment  worksheet  is  large  and  a  completed  assessment  is  in  Appendix  D. 
Figure  22  shows  a  portion  of  the  completed  assessment. 


System 

Anti  Air 

Defense 

Mission 

Weight 

Anti  Surface 

Warfare 

Mission 

Weight 

Modernization 

Weight 

3D  Volume  Search  Radar  (SPS-48) 

Necessary 

0.75 

Non-Critical 

0.25 

Improve  Capability 

1 

2D  Volume  Search  Radar  (SPS-49) 

Necessary 

0.75 

Non-Critical 

0.25 

No  Change 

0 

Identification  Friend  or  Foe  (UPX-29) 

Like  to  have 

0.5 

Like  to  have 

0.5 

No  Change 

0 

High  Fidelity  Radar  RADAR  (SPQ-9B) 

Critical 

1 

Critical 

1 

Improve  Performance 

0.5 

Navigation  Radar  (SPS-73) 

Non-Critical 

0.25 

Non-Critical 

0.25 

Improve  Performance 

0.5 

Surface  Search  Radar (SPS-67) 

Non-Critical 

0.25 

Critical 

1 

Improve  Infrastructures 

0.25 

Navigation  Data  (WSN-7) 

Necessary 

0.75 

Necessary 

0.75 

No  Change 

0 

MK38  MGS 

Non-Critical 

0.25 

Necessary 

0.75 

Replace  with  New 

1 

System 

AD  Impact 

Score 

Non- 

SUW  Impact  Critical 
Score  Weight 

3D  Volume  Search  Radar  (SPS  48) 

0.75 

0 

0 

2D  Volume  Search  Radar  (SPS-49) 

0 

0 

0 

Identification  Friend  or  Foe  (UPX-29) 

0 

0 

0 

High  Fidelity  Radar  RADAR  (SPQ-9B) 

0.5 

0.5 

0 

Navigation  Radar  (SPS-73) 

0 

0 

0.125 

Surface  Search  Radar  (SPS-67) 

0 

0.25 

0 

Navigation  Data  (WSN-7) 

0 

0 

0 

MK38MGS 

0 

0.75 

0 

Figure  2 1 .  Portion  of  Completed  Significance  Assessment 


The  ship  used  for  the  assessment  in  Figure  21  has  29  total  combat/weapon 
systems  onboard.  The  AD  Impact  Score,  SUW  Impact  Score,  and  Non-Critical  Weight 
columns  are  summed  and  this  sum  is  the  value  used  in  the  Determination  Dashboard. 

2.  Risk  to  Maintenance  Readiness 

The  methodology  described  in  Chapter  III  determined  the  best  method  for 
assessing  the  risk  to  maintenance  readiness  is  use  a  binary  system.  The  first  step  is  to 
create  the  assessment  workspace.  The  logical  place  to  conduct  the  assessment  is  on  the 
same  worksheet  the  Significance  Assessment  is  conducted  because  the  risk  assessment  is 
analyzing  systems  that  are  being  modified  during  the  availability.  Each  system  that  is 
being  altered  is  copied  from  the  list  of  systems  in  the  Significance  Assessment  and  listed 
in  an  area  below  the  Significance  Assessment.  The  second  step  is  to  label  columns  with 
the  risk  factors  from  the  methodology.  The  risk  factors  are: 

•  Hardware  Change 

•  Software  Change 
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•  PMS  Impacted 

•  Technical  Manuals  Impacted 

•  Part  Supply  Impacted 

•  Training  Impacted 

•  Certification  Required 

•  Test  Equipment  Impacted 

•  Operating  Sequencing  System  Impacted 

•  Detect-to-Engage  Path  Impacted 

The  third  step  is  to  collect  all  the  SCDs  applicable  to  the  ship  from  the  NDE.  In 
most  cases,  when  there  is  a  software  change  and  a  hardware  change  on  the  same  system, 
there  will  be  two  separate  SCDs.  The  SCD  sections  from  the  methodology  are  reviewed 
and  if  the  SCD  indicates  the  change  will  impact  a  risk  factor,  the  fourth  step  is  to  enter  a 
1  in  the  column.  If  it  is  not  being  affected,  a  zero  is  entered.  The  final  step  is  to  sum  the 
values  for  each  system  to  get  a  risk  score  for  each  system  and  then  sum  the  risk  scores  to 
get  an  overall  risk  to  maintenance  readiness  score  which  is  used  in  the  Detennination 
Dashboard.  Figure  23  shows  a  completed  assessment  (broken  up  for  display  purposes). 
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Impacted  System 

Hardware  Change 

Software 

Change 

PMS  Impacted 

Tech  Manuals 
Impacted 

Part  Supply  Impacted 

3D  Volume  Search  Radar  (SPS-48) 

1 

1 

1 

1 

1 

High  Fidel  ity  Radar  RADAR  (SPQ  98) 

0 

1 

0 

1 

0 

Navigation  Radar  (SPS-73) 

1 

1 

1 

1 

1 

Surface  Search  Radar  (SPS-67) 

1 

0 

1 

1 

1 

MK38  MGS 

1 

1 

1 

1 

1 

Critical  Navigation  Distribution  (NCDS) 

0 

1 

0 

1 

0 

Command  and  Control  (SSDS) 

1 

1 

1 

1 

1 

BFTT 

0 

1 

0 

1 

0 

TSSS 

1 

1 

1 

1 

1 

Electronic  Signal  Detection  (SLQ-32) 

1 

1 

1 

1 

1 

Self  Defense  (RAM) 

1 

1 

1 

1 

1 

Point  Defense  (CIWS) 

1 

1 

1 

1 

1 

CDLMS 

0 

1 

0 

1 

0 

Training 

Impacted 

Certifications 

Required 

Test  Equipment 
Impacted 

Operating 

Sequence 

Systems 

Impacted 

DTE 

Impacted 

Sum 

1 

1 

0 

1 

1 

9 

1 

1 

0 

0 

1 

5 

1 

1 

0 

0 

0 

7 

0 

1 

0 

0 

1 

6 

1 

1 

0 

1 

1 

9 

0 

1 

0 

0 

0 

3 

1 

1 

0 

0 

1 

8 

1 

1 

0 

0 

0 

4 

1 

1 

0 

0 

0 

7 

1 

1 

0 

1 

1 

9 

1 

1 

0 

1 

1 

9 

1 

1 

0 

0 

1 

8 

1 

1 

0 

0 

0 

4 

6.77 

Figure  22.  Completed  Risk  to  Maintenance  Readiness  Assessment 


C.  CSSQT  DETERMINATION  MATRIX 

The  final  piece  of  the  CSSQT  Determination  Dashboard  is  the  Detennination 
Matrix.  When  the  all  information  is  gathered  and  inputted  for  each  ship,  the  final  overall 
assessment  for  the  fiscal  year  will  look  like  Figure  24. 
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Availability  Data 

Significance  Assessment 

Ship 

Avail 

Avail  Start 

Avail  End 

AD 

Impact 

Score 

SUW 

Impact 

Score 

Non-Crit 

Weight 

significance 

Score 

CVN  68 

PIA3 

May- 14 

Feb- 15 

5.13 

4.13 

0.31 

4.94 

CVN77 

PIA1 

Apr-15 

Oct- 15 

5.63 

3.88 

0.69 

5.44 

SRA 

1.00 

1.00 

0.00 

1.00 

5.00 

4.75 

0.69 

5.56 

2.50 

3.13 

0.00 

2.81 

LSD  50 

EDPMA 

Jan-14 

Jan-15 

6.25 

6.00 

0.75 

6.88 

LSD  52 

EDPMA 

Feb- 14 

Feb-15 

6.25 

5.13 

0.63 

6.31 

Sep- 14 

Mar- 15 

5.50 

3.00 

5.50 

CVN  76 

Oct- 14 

Apr-15 

5.00 

4.00 

2.00 

6.50 

LHD  8 

PMA1-2 

Feb-15 

Aug- 15 

3.75 

3.50 

3.75 

Risk  Assessment 

Avail 

Length 

(Months) 

Total 

Systems 

Total 

Systems 

Changed 

% 

Systems 

Changed 

Last 

CSSQT 

Delta 

(Years) 

Operational 

Readiness 

Score 

Maintenance 

Readiness 

Score 

Safety 

Score 

Risk 

Likelihood 

9.00 

29 

13 

45% 

2007 

8 

3.70 

6.77 

10 

6.82 

6.00 

31 

13 

6 

2.52 

7.08 

5 

4.86 

0.25 

18 

2 

29 

1.11 

5.5 

5 

3.87 

0.33 

20 

14 

26 

7.00 

6.57 

10 

7.86 

0.25 

19 

7 

22 

3.68 

5.71 

5 

4.80 

1.00 

20 

15 

1995 

20 

7.50 

6.80 

10 

8.10 

1.00 

20 

14 

1998 

17 

7.00 

6.80 

10 

7.93 

0.50 

31 

16 

52% 

2007 

8 

3.16 

7.00 

10 

6.72 

0.50 

32 

16 

50% 

2008 

7 

2.69 

10 

6.48 

6.00 

29 

8 

28% 

2009 

6 

1.66 

7.5 

10 

Figure  23.  Completed  FY15  CSSQT  Determination  Dashboard 


The  Significance  Score  and  Risk  Likelihood  (highlighted  with  dark  boxes)  are  the 
values  plotted  on  the  Determination  Matrix  chart  shown  in  Figure  25. 
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Figure  24.  FY15  CSSQT  Determination  Matrix 


The  matrix  is  divided  into  four  quadrants  and  each  quadrant  is  labeled  with  a 
recommendation.  If  a  ship  is  plotted  in  quadrant  one  a  CSSQT  is  recommended  because 
there  are  significant  changes  being  made  to  the  ship,  but  the  likelihood  that  ship’s 
readiness  will  be  at  risk  is  low.  A  ship  in  quadrant  two  has  significant  changes  being 
made  and  the  likelihood  that  the  changes  will  impact  ship’s  readiness  is  high;  therefore,  it 
is  recommended  that  a  CSSQT  is  required  for  that  ship.  In  quadrant  three  the  changes  are 
not  significant  but  the  likelihood  that  the  changes  will  impact  ship  readiness  is  high,  so 
the  ship  is  recommended  to  conduct  a  CSSQT.  A  ship  in  quadrant  four  does  not  have 
significant  amount  of  changes  and  the  changes  have  less  likelihood  to  impact  ship’s 
readiness  and  it  is  recommended  that  a  ship  in  the  fourth  quadrant  does  not  conduct  a 
CSSQT.  Ships  in  quadrants  one  and  four  are  recommended  for  CSSQT  which  means  if 
funding  and  resources  are  available  then  these  ships  should  be  designated  to  perfonn  a 
CSSQT.  Ships  in  quadrant  two  would  benefit  the  most  from  a  CSSQT  and  it  those  ships 
should  be  the  first  to  be  allocated  funding  and  resources. 
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This  chapter  used  the  methodology  and  the  results  of  the  review  of  current  work 
discussed  in  Chapter  III  to  develop  a  better  way  to  assess  the  impact  a  maintenance 
availability  has  on  a  ship’s  readiness.  The  CSSQT  Determination  Dashboard  was 
designed  to  put  all  the  results  of  the  assessment  in  a  central  location.  The  CSSQT 
Decision  Tool  process  used  by  NSWC  PHD  was  redesigned  and  the  name  was  changed 
to  Significance  Assessment  for  clarity  about  the  purpose  of  the  tool.  The  risk  assessment 
process  was  completely  changed  to  align  with  NAVSEAINST  9093. 1C  and  to  remove 
subjective  assessments.  Finally,  the  Detennination  Matrix  was  introduced  to  graphically 
show  the  impact  to  ship’s  readiness.  The  next  and  final  chapter  will  discuss  conclusions 
and  recommendations  for  further  research. 
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V.  CONCLUSION  AND  RECOMMENDATIONS 


The  purpose  of  this  thesis  is  to  develop  a  process  to  recommend  ships  for  CSSQT 
by  detennining  the  impact  an  overhaul/modernization  has  on  a  ship’s  readiness.  This  was 
accomplished  by  analyzing  a  process  used  by  NSWC  PHD  to  recommend  ships  for 
CSSQT  and  determine  if  the  process  can  be  improved.  The  ultimate  goal  of  the  process  is 
to  determine  the  impact  on  a  ship’s  readiness  based  on  how  significant  the  changes  are 
and  the  likelihood  the  changes  will  impact  readiness.  The  results  of  the  process  aid 
decision-makers  in  determining  if  a  CSSQT  is  required  for  a  ship  post-availability.  The 
thesis  presented  background  information  about  CSSQT  and  the  current  process  used  by 
NSWC  PHD.  A  majority  of  the  work  focused  on  a  creating  a  methodology  for  reviewing 
the  current  process  and  an  approach  to  correct  issues  discovered  during  the  review. 
Finally,  the  work  resulted  in  an  assessment  process  that  quantifies  the  impact  a  major 
overhaul/modemization  has  on  a  ship’s  readiness  in  a  valid  and  objective  way. 

A.  SUMMARY  OF  FINDINGS 

The  review  of  the  current  tools  used  by  NSWC  PHD  was  intended  to  investigate 
the  processes  used  for  flaws  and  issues.  The  analysis  a  determined  that  the  method  for 
determining  significance  was  flawed  for  the  following  reasons: 

•  The  name  of  tool  does  not  accurately  reflect  the  purpose  of  the  tool 

•  There  is  an  inaccurate  accountability  of  all  combat/weapon  systems 

•  The  significance  calculation  result  is  ambiguous 

•  The  significance  calculation  does  not  account  for  non-critical  systems 

•  A  completed  assessment  can  result  in  a  spreadsheet  that  contains  empty 
cells  and  cells  that  contain  FALSE  as  an  entry 

The  risk  assessment  review  found  the  following  issues  with  the  proposed  process: 

•  The  risk  areas  do  not  align  with  the  objectives  outlined  in  NAVSEAINST 
9093. 1C 

•  Risk  factors  applied  to  more  than  one  risk  area  which  made  the  assessment 
complicated 

•  A  large  number  of  risk  factors  required  subjective  input  from  outside 
sources 
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•  The  output  of  the  assessment  was  ambiguous 

The  byproducts  of  the  review  were  a  better  understanding  of  the  purpose  of  the 
assessment  and  a  new  methodology  for  conducting  the  assessment.  The  methodology 
determined  the  key  to  the  assessment  is  the  result  has  to  be  on  a  measurable  scale.  That  is 
how  ambiguity  is  removed  from  the  output.  By  setting  a  scale  and  putting  the  result  on 
that  scale,  there  is  more  clarity  in  what  the  result  means.  The  methodology  used  a  0  to  10 
scale  for  both  significance  and  risk  likelihood.  The  higher  the  result  of  the  assessment  the 
more  significant  the  changes  are  and  the  greater  likelihood  the  changes  will  impact  ship’s 
readiness.  Using  the  same  scale  for  each  assessment  also  allows  the  results  to  be  plotted 
on  a  graph  for  better  overall  understanding  and  comparison  with  other  ship  being 
assessed. 

The  approach  developed  in  this  thesis  is  not  perfect,  but  it  does  present  a  way  to 
assess  the  impact  a  major  overhaul/modernization  will  have  on  a  ship.  The  assessment  is 
completely  quantitative  and  will  yield  dependable  results  when  repeated  for  multiple 
ships.  It  also  gives  decision-makers  a  level  of  detail  that  does  not  overwhelm  but  enough 
to  make  a  sound  decision  about  moving  forward  or  not  in  designating  a  ship  for  CSSQT. 

B.  RECOMMENDATIONS  FOR  FURTHER  STUDY 

Assessments  that  include  risk  are  better  when  there  is  historical  data  available. 
The  next  logical  step  in  for  assessing  the  impact  a  maintenance  availability  is  to  collect 
data  about  the  ship’s  performance  post-availability.  The  data  needs  to  include  ships  that 
conduct  a  CSSQT  and  those  that  do  not  to  determine  if  there  is  a  correlation  between 
ship’s  readiness  and  the  conduction  of  a  CSSQT.  There  could  be  other  correlations,  such 
as  length  of  availability  and  ship’s  readiness  or  number  of  systems  changed  and  ship’s 
readiness.  The  result  would  be  thresholds  that  each  ship  could  be  compared  to.  For 
example,  a  ship  that  is  conducting  a  12  month  availability  and  is  having  75%  of  its 
combat/weapon  systems  altered  should  conduct  a  CSSQT  because  historical  data  shows 
that  ships  having  these  characteristics,  and  did  not  conduct  a  CSSQT,  are  more  likely  to 
be  delayed  in  the  training  cycle.  The  research  would  require  determining  what  data  to 
collect  and  which  metrics  should  be  analyzed. 
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This  research  was  only  conducted  on  three  classes  of  surface  ships  but  CSSQT  is 
conducted  on  all  surface  ships.  The  tool  needs  to  be  expanded  to  assess  other  classes  of 
ships  so  there  is  a  common  process  for  designating  all  surface  ship  for  CSSQT.  This 
could  be  accomplished  by  enhancing  the  CSSQT  Detennination  Process  by  using 
computer  software  that  allows  users  to  easily  complete  an  assessment.  Using  a  software 
program  could  allow  users  to  select  the  ship  class  and  it  automatically  generates  an 
interface  to  input  required  information.  The  use  of  VBA  is  also  a  potential  way  to  make 
the  process  simpler. 

The  final  area  that  requires  further  research  is  the  periodicity  of  CSSQT.  CSSQT 
is  a  valuable  evolutions  that  benefits  both  the  crew  and  the  ship.  For  some  ships,  a 
CSSQT  is  only  conducted  once  in  the  entire  life  of  the  ship.  An  analysis  of  the 
importance  of  CSSQT  should  be  conducted  to  determine  if  a  CSSQT  should  be 
conducted  on  a  more  routine  schedule. 
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Ship:  USS  Nimitz  (CVN-68)  Avail:  FY14  DPIA3 

Significance  Assessment 
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Total  Systems  29 

Systems  Changed  13 

%  Changed  45% 
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